2020-03-08T20:45:11.222 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-08T20:45:11.223 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.LvxGEa.tmp -- --verbose --lock --description rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-comp-zlib.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_8.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} --name sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152 2020-03-08T20:45:11.263 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-08T20:45:11.326 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152 branch: wip-sage4-testing-2020-03-08-0938 description: rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-comp-zlib.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_8.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} email: null first_in_suite: false job_id: '4838152' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-sage4-testing-2020-03-08-0938 ceph: conf: global: mon client directed command retry: 5 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_async_recovery_min_cost: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-sage4-testing-2020-03-08-0938 sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b owner: scheduled_sage@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b suite: rados suite_branch: wip-sage4-testing-2020-03-08-0938 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27946 2020-03-08T20:45:11.326 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa; will attempt to use it 2020-03-08T20:45:11.327 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks 2020-03-08T20:45:11.327 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-08T20:45:11.329 INFO:teuthology.task.internal:Checking packages... 2020-03-08T20:45:11.343 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '9eb9490ed0531fd33292066429bf9db55b9c1d3b' 2020-03-08T20:45:11.344 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-08T20:45:11.344 INFO:teuthology.packaging:ref: None 2020-03-08T20:45:11.344 INFO:teuthology.packaging:tag: None 2020-03-08T20:45:11.344 INFO:teuthology.packaging:branch: wip-sage4-testing-2020-03-08-0938 2020-03-08T20:45:11.344 INFO:teuthology.packaging:sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-08T20:45:11.344 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage4-testing-2020-03-08-0938 2020-03-08T20:45:11.507 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1904.g9eb9490 2020-03-08T20:45:11.507 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-08T20:45:11.518 INFO:teuthology.task.internal:no buildpackages task found 2020-03-08T20:45:11.518 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-08T20:45:11.531 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-08T20:45:11.531 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-08T20:45:11.584 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:45:21.604 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:45:31.619 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:45:41.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:45:51.657 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:46:01.669 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:46:11.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:46:21.705 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:46:31.724 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:46:41.740 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:46:51.758 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:47:01.776 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:47:11.794 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:47:21.812 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:47:31.826 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:47:41.841 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:47:51.858 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:48:01.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:48:11.885 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:48:21.904 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:48:31.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:48:41.935 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:48:51.952 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:49:01.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:49:11.988 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:49:22.002 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:49:32.018 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:49:42.034 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:49:52.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:50:02.067 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:50:12.083 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:50:22.101 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:50:32.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:50:42.140 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:50:52.155 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:51:02.170 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:51:12.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:51:22.229 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:51:32.248 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:51:42.268 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T20:51:52.283 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T20:52:02.301 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T20:52:12.319 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T20:52:22.331 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T20:52:32.351 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:52:42.367 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:52:52.385 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:53:02.403 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:53:12.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:53:22.438 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:53:32.457 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:53:42.473 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:53:52.492 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:54:02.506 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:54:12.525 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:54:22.543 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:54:32.559 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:54:42.574 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:54:52.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:55:02.606 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:55:12.627 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:55:22.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:55:32.666 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T20:55:42.685 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:55:52.702 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:56:02.719 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:56:12.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:56:22.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:56:32.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:56:42.783 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:56:52.799 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:57:02.816 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:57:12.837 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:57:22.859 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:57:32.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:57:42.898 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:57:52.917 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:58:02.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T20:58:12.951 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:58:22.966 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:58:32.986 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:58:43.004 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:58:53.022 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:59:03.032 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:59:13.050 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:59:23.062 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:59:33.075 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:59:43.092 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T20:59:53.108 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:00:03.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:00:13.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:00:23.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:00:33.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:00:43.206 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:00:53.223 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:01:03.240 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:01:13.258 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:01:23.274 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:01:33.294 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:01:43.312 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:01:53.332 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:02:03.349 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:02:13.370 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:02:23.387 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:02:33.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:02:43.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:02:53.434 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:03:03.452 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:03:13.467 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:03:23.485 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:03:33.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:03:43.519 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:03:53.538 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:04:03.554 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:04:13.570 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:04:23.589 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:04:33.607 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:04:43.622 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:04:53.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:05:03.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:05:13.667 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:05:23.685 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:05:33.703 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:05:43.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:05:53.728 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:06:03.742 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:06:13.759 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:06:23.778 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:06:33.797 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:06:43.814 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:06:53.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:07:03.844 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:07:13.854 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:07:23.872 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:07:33.891 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:07:43.909 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:07:53.920 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)... 2020-03-08T21:08:03.931 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:08:13.946 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:08:23.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:08:33.978 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:08:43.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:08:54.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:09:04.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:09:14.045 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:09:24.057 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:09:34.074 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:09:44.091 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:09:54.108 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:10:04.119 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:10:14.134 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:10:24.152 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:10:34.162 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:10:44.178 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:10:54.195 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:11:04.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:11:14.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:11:24.241 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:11:34.263 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:11:44.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:11:54.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:12:04.321 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:12:14.341 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:12:24.384 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:12:34.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:12:44.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:12:54.434 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:13:04.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:13:14.467 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:13:24.483 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:13:34.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:13:44.520 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:13:54.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:14:04.549 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:14:14.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:14:24.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:14:34.594 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:14:44.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:14:54.624 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:15:04.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:15:14.681 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:15:24.690 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:15:34.706 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:15:44.724 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:15:54.738 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:16:04.755 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:16:14.771 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:16:24.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:16:34.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:16:44.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:16:54.842 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:17:04.858 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:17:14.876 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:17:24.895 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:17:34.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:17:44.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:17:54.946 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:18:04.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:18:14.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:18:24.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:18:35.013 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:18:45.032 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:18:55.043 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:19:05.058 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:19:15.080 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:19:25.098 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:19:35.108 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:19:45.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:19:55.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:20:05.159 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:20:15.178 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:20:25.196 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:20:35.212 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:20:45.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:20:55.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:21:05.261 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:21:15.273 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:21:25.291 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:21:35.305 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:21:45.323 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:21:55.341 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:22:05.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:22:15.375 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:22:25.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:22:35.411 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:22:45.459 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:22:55.477 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:23:05.495 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:23:15.512 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:23:25.530 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:23:35.548 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:23:45.566 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:23:55.586 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:24:05.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:24:15.618 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:24:25.635 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:24:35.651 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:24:45.666 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:24:55.684 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:25:05.702 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:25:15.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:25:25.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:25:35.751 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)... 2020-03-08T21:25:45.766 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:25:55.783 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:26:05.799 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:26:15.816 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:26:25.832 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:26:35.851 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:26:45.866 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:26:55.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:27:05.902 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:27:15.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:27:25.937 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:27:35.955 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:27:45.971 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:27:55.996 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:28:06.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:28:16.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:28:26.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:28:36.059 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:28:46.077 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:28:56.095 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:29:06.112 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:29:16.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:29:26.149 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:29:36.199 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:29:46.216 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:29:56.234 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:30:06.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:30:16.265 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:30:26.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:30:36.301 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:30:46.318 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:30:56.339 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:31:06.350 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:31:16.368 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:31:26.382 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:31:36.401 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:31:46.419 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:31:56.438 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:32:06.454 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:32:16.464 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:32:26.483 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:32:36.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:32:46.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:32:56.527 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:33:06.543 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:33:16.562 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:33:26.582 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:33:36.599 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:33:46.616 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:33:56.630 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:34:06.648 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:34:16.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:34:26.679 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:34:36.692 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:34:46.708 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:34:56.723 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:35:06.739 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:35:16.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:35:26.775 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:35:36.786 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:35:46.801 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:35:56.819 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:36:06.831 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:36:16.847 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:36:26.863 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:36:36.877 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:36:46.895 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)... 2020-03-08T21:36:56.910 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:37:06.928 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:37:16.937 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:37:26.959 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:37:36.976 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:37:46.989 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:37:57.007 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:38:07.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:38:17.045 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:38:27.060 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:38:37.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:38:47.095 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:38:57.110 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:39:07.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:39:17.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:39:27.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:39:37.178 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:39:47.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:39:57.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:40:07.222 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:40:17.239 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:40:27.255 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:40:37.273 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:40:47.291 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:40:57.305 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:41:07.324 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:41:17.341 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:41:27.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:41:37.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:41:47.394 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:41:57.411 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:42:07.432 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:42:17.452 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:42:27.471 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:42:37.488 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:42:47.507 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:42:57.517 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:43:07.535 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:43:17.553 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:43:27.571 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:43:37.588 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:43:47.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:43:57.618 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:44:07.636 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:44:17.658 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:44:27.672 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:44:37.691 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:44:47.707 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:44:57.721 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:45:07.737 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:45:17.755 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:45:27.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:45:37.790 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:45:47.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:45:57.854 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:46:07.870 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:46:17.889 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:46:27.907 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T21:46:37.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:46:47.946 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T21:46:57.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:47:07.981 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:47:17.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:47:28.013 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:47:38.035 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:47:48.051 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:47:58.068 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:48:08.087 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:48:18.103 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:48:28.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:48:38.141 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:48:48.156 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:48:58.166 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:49:08.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:49:18.203 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:49:28.219 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:49:38.237 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:49:48.252 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:49:58.267 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:50:08.283 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:50:18.301 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:50:28.313 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:50:38.332 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:50:48.353 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:50:58.371 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:51:08.389 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:51:18.399 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:51:28.417 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:51:38.436 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:51:48.454 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:51:58.469 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:52:08.487 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:52:18.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:52:28.523 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:52:38.542 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:52:48.556 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:52:58.574 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:53:08.593 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:53:18.611 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)... 2020-03-08T21:53:28.626 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:53:38.641 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:53:48.654 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:53:58.671 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:54:08.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:54:18.707 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:54:28.725 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:54:38.743 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:54:48.761 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:54:58.779 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:55:08.795 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:55:18.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:55:28.828 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:55:38.841 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:55:48.858 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:55:58.876 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:56:08.931 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:56:18.948 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T21:56:28.959 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:56:38.968 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:56:48.981 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:56:58.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:57:09.015 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:57:19.034 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:57:29.052 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:57:39.065 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:57:49.086 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:57:59.099 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:58:09.114 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:58:19.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:58:29.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:58:39.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:58:49.173 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:58:59.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:59:09.209 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:59:19.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T21:59:29.242 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:59:39.258 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:59:49.274 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T21:59:59.290 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:00:09.307 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:00:19.325 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:00:29.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:00:39.363 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:00:49.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:00:59.394 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:01:09.410 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:01:19.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:01:29.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:01:39.455 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:01:49.469 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:01:59.486 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:02:09.499 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:02:19.515 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:02:29.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:02:39.547 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:02:49.561 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:02:59.579 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:03:09.597 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:03:19.615 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:03:29.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:03:39.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:03:49.665 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:03:59.676 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:04:09.694 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:04:19.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:04:29.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:04:39.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:04:49.764 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:04:59.784 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:05:09.798 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:05:19.816 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:05:29.832 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:05:39.849 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:05:49.867 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:05:59.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:06:09.902 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:06:19.920 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:06:29.940 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:06:39.956 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:06:49.973 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:06:59.990 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:07:10.006 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:07:20.025 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:07:30.037 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:07:40.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:07:50.069 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:08:00.086 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:08:10.104 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:08:20.119 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:08:30.137 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:08:40.152 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:08:50.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:09:00.177 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:09:10.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:09:20.232 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:09:30.242 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:09:40.259 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:09:50.277 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:10:00.295 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:10:10.311 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:10:20.321 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:10:30.337 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:10:40.354 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:10:50.369 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:11:00.387 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:11:10.404 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:11:20.422 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:11:30.441 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:11:40.454 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:11:50.479 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:12:00.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:12:10.516 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:12:20.530 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:12:30.545 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:12:40.555 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:12:50.573 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:13:00.621 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:13:10.637 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:13:20.655 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:13:30.673 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:13:40.690 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:13:50.707 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:14:00.723 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:14:10.741 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:14:20.759 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:14:30.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:14:40.790 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:14:50.803 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:15:00.820 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:15:10.839 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:15:20.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:15:30.873 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:15:40.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:15:50.909 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:16:00.926 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:16:10.944 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:16:20.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:16:30.974 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:16:40.991 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:16:51.006 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:17:01.024 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:17:11.041 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:17:21.057 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:17:31.076 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:17:41.091 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:17:51.106 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:18:01.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:18:11.139 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:18:21.158 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:18:31.174 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:18:41.203 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:18:51.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:19:01.234 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:19:11.250 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:19:21.266 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:19:31.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:19:41.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:19:51.321 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:20:01.338 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:20:11.354 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:20:21.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:20:31.390 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:20:41.406 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:20:51.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:21:01.441 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:21:11.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:21:21.489 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:21:31.505 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:21:41.522 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:21:51.541 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:22:01.553 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:22:11.571 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:22:21.583 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:22:31.598 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:22:41.611 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:22:51.627 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:23:01.643 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:23:11.661 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:23:21.678 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:23:31.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:23:41.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:23:51.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:24:01.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:24:11.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:24:21.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:24:31.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:24:41.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:24:51.822 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T22:25:01.840 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:25:11.856 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:25:21.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:25:31.888 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:25:41.898 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:25:51.911 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:26:01.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:26:11.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:26:21.995 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:26:32.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:26:42.032 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:26:52.055 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:27:02.073 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:27:12.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:27:22.107 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:27:32.123 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:27:42.138 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:27:52.154 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:28:02.170 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:28:12.186 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:28:22.205 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:28:32.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:28:42.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:28:52.258 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:29:02.274 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:29:12.290 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:29:22.301 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:29:32.319 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:29:42.334 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:29:52.351 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:30:02.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:30:12.388 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:30:22.407 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:30:32.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:30:42.438 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:30:52.454 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:31:02.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:31:12.490 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:31:22.506 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:31:32.524 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:31:42.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:31:52.551 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:32:02.566 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:32:12.575 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:32:22.588 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:32:32.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:32:42.617 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:32:52.628 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:33:02.646 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:33:12.657 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:33:22.675 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:33:32.688 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:33:42.705 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:33:52.719 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:34:02.738 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:34:12.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:34:22.771 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:34:32.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:34:42.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:34:52.824 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:35:02.843 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:35:12.860 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:35:22.876 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:35:32.891 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:35:42.907 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:35:52.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:36:02.933 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:36:12.951 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:36:22.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:36:32.987 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:36:43.003 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:36:53.020 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:37:03.038 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:37:13.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:37:23.070 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:37:33.089 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:37:43.098 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:37:53.116 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:38:03.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:38:13.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:38:23.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:38:33.181 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:38:43.193 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:38:53.211 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:39:03.226 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:39:13.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:39:23.258 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:39:33.275 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:39:43.288 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:39:53.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:40:03.322 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T22:40:13.332 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T22:40:23.350 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:40:33.364 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:40:43.381 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:40:53.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:41:03.407 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:41:13.421 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:41:23.439 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:41:33.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:41:43.466 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:41:53.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:42:03.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:42:13.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:42:23.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T22:42:33.544 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:42:43.559 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:42:53.579 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:43:03.595 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:43:13.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:43:23.632 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:43:33.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:43:43.664 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:43:53.682 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:44:03.696 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:44:13.711 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:44:23.728 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:44:33.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:44:43.761 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:44:53.778 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:45:03.796 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:45:13.815 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:45:23.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:45:33.843 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:45:43.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:45:53.874 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:46:03.889 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:46:13.910 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:46:23.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:46:33.943 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:46:43.958 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:46:53.976 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:47:03.991 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:47:14.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:47:24.093 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:47:34.111 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:47:44.129 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:47:54.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:48:04.162 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:48:14.180 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:48:24.199 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:48:34.216 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:48:44.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:48:54.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:49:04.265 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:49:14.280 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:49:24.298 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:49:34.314 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:49:44.331 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:49:54.346 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:50:04.365 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:50:14.383 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:50:24.396 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:50:34.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:50:44.433 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:50:54.451 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:51:04.469 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:51:14.486 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:51:24.496 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:51:34.514 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:51:44.530 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:51:54.546 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:52:04.562 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:52:14.580 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:52:24.599 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:52:34.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:52:44.632 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:52:54.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:53:04.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:53:14.678 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:53:24.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:53:34.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:53:44.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:53:54.722 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:54:04.738 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:54:14.754 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:54:24.773 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:54:34.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:54:44.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:54:54.824 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:55:04.839 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:55:14.859 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:55:24.868 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:55:34.884 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:55:44.899 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:55:54.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:56:04.932 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:56:14.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:56:24.967 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:56:34.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:56:45.002 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:56:55.019 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:57:05.034 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:57:15.044 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:57:25.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:57:35.078 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:57:45.097 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:57:55.115 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:58:05.132 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:58:15.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:58:25.158 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:58:35.176 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:58:45.191 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:58:55.206 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:59:05.226 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:59:15.242 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T22:59:25.259 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:59:35.277 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:59:45.298 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T22:59:55.311 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:00:05.326 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:00:15.344 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:00:25.362 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:00:35.380 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:00:45.400 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:00:55.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:01:05.425 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:01:15.435 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:01:25.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:01:35.469 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:01:45.487 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:01:55.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:02:05.520 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:02:15.538 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:02:25.552 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:02:35.571 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:02:45.582 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:02:55.596 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:03:05.610 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:03:15.629 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:03:25.648 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:03:35.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:03:45.676 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:03:55.694 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:04:05.712 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:04:15.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:04:25.738 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:04:35.754 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:04:45.766 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:04:55.783 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:05:05.792 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:05:15.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:05:25.827 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:05:35.836 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:05:45.846 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:05:55.864 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:06:05.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:06:15.898 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:06:25.916 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:06:35.933 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:06:45.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:06:55.966 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:07:05.984 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:07:16.001 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:07:26.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:07:36.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:07:46.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:07:56.062 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:08:06.078 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:08:16.094 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:08:26.113 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:08:36.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:08:46.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:08:56.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:09:06.176 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:09:16.195 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:09:26.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:09:36.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:09:46.242 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:09:56.254 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:10:06.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:10:16.291 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:10:26.309 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:10:36.327 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:10:46.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:10:56.360 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:11:06.374 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:11:16.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:11:26.410 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:11:36.429 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:11:46.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:11:56.461 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:12:06.479 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:12:16.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:12:26.514 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:12:36.529 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:12:46.546 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:12:56.562 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:13:06.578 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:13:16.597 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:13:26.615 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:13:36.633 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:13:46.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:13:56.669 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:14:06.682 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:14:16.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:14:26.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:14:36.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:14:46.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:14:56.767 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:15:06.780 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:15:16.790 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:15:26.807 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:15:36.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:15:46.843 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:15:56.860 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:16:06.877 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:16:16.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:16:26.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:16:36.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:16:46.957 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:16:56.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:17:06.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:17:17.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:17:27.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:17:37.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:17:47.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:17:57.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:18:07.070 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T23:18:17.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:18:27.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:18:37.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:18:47.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:18:57.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:19:07.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:19:17.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:19:27.215 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:19:37.233 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:19:47.248 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:19:57.261 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:20:07.280 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:20:17.293 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:20:27.311 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:20:37.329 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:20:47.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:20:57.365 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:21:07.384 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:21:17.404 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:21:27.423 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:21:37.439 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:21:47.455 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:21:57.471 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:22:07.485 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:22:17.497 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:22:27.514 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:22:37.524 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:22:47.544 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:22:57.574 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:23:07.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:23:17.611 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:23:27.629 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:23:37.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:23:47.662 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:23:57.681 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:24:07.700 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:24:17.714 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:24:27.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:24:37.748 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:24:47.766 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:24:57.784 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:25:07.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:25:17.821 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:25:27.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:25:37.848 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:25:47.867 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:25:57.883 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:26:07.897 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:26:17.916 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:26:27.932 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:26:37.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:26:47.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:26:57.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:27:07.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:27:18.016 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:27:28.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:27:38.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:27:48.053 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:27:58.066 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:28:08.083 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:28:18.102 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:28:28.114 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-03-08T23:28:38.130 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:28:48.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:28:58.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:29:08.180 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:29:18.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T23:29:28.216 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:29:38.230 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:29:48.247 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:29:58.265 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:30:08.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:30:18.296 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:30:28.315 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:30:38.331 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:30:48.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:30:58.362 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:31:08.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:31:18.394 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:31:28.411 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:31:38.428 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:31:48.490 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:31:58.505 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:32:08.522 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:32:18.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:32:28.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:32:38.575 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:32:48.588 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:32:58.606 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:33:08.623 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:33:18.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:33:28.655 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:33:38.668 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:33:48.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:33:58.705 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:34:08.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:34:18.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:34:28.764 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:34:38.779 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:34:48.794 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:34:58.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:35:08.827 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:35:18.842 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:35:28.857 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:35:38.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:35:48.890 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:35:58.908 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:36:08.918 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:36:18.936 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:36:28.951 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:36:38.965 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:36:48.984 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:36:58.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:37:09.016 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:37:19.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:37:29.036 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:37:39.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:37:49.065 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:37:59.083 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:38:09.097 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:38:19.107 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:38:29.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:38:39.138 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:38:49.156 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:38:59.175 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:39:09.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:39:19.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:39:29.223 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:39:39.239 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:39:49.254 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:39:59.270 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:40:09.286 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:40:19.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:40:29.319 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:40:39.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:40:49.353 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:40:59.370 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:41:09.387 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:41:19.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:41:29.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:41:39.433 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:41:49.448 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:41:59.462 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:42:09.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:42:19.490 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:42:29.508 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:42:39.522 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:42:49.540 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:42:59.550 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:43:09.566 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:43:19.585 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:43:29.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:43:39.618 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:43:49.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:43:59.645 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:44:09.662 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:44:19.680 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:44:29.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:44:39.712 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:44:49.733 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:44:59.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:45:09.769 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:45:19.787 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:45:29.803 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:45:39.821 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:45:49.840 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:45:59.854 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:46:09.867 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:46:19.883 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:46:29.901 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:46:39.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:46:49.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:46:59.951 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:47:09.967 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:47:19.982 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:47:30.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:47:40.010 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:47:50.024 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:48:00.043 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:48:10.060 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:48:20.075 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:48:30.091 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:48:40.108 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:48:50.125 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:49:00.142 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:49:10.161 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:49:20.179 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:49:30.195 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:49:40.213 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:49:50.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:50:00.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:50:10.262 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:50:20.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:50:30.293 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:50:40.312 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:50:50.330 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:51:00.346 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:51:10.363 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:51:20.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:51:30.394 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:51:40.410 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:51:50.428 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:52:00.446 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:52:10.455 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:52:20.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:52:30.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:52:40.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:52:50.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:53:00.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:53:10.550 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:53:20.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:53:30.584 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:53:40.598 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:53:50.614 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:54:00.624 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:54:10.641 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:54:20.652 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:54:30.666 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:54:40.684 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:54:50.702 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:55:00.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:55:10.737 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-03-08T23:55:20.755 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:55:30.773 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:55:40.791 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:55:50.804 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:56:00.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:56:10.838 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:56:20.857 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:56:30.873 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-08T23:56:40.890 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:56:50.900 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:57:00.913 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:57:10.923 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:57:20.942 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:57:30.957 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:57:40.974 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:57:50.990 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:58:01.006 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-03-08T23:58:11.018 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:58:21.036 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:58:31.049 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:58:41.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:58:51.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:59:01.100 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:59:11.114 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:59:21.132 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:59:31.150 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:59:41.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-08T23:59:51.174 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:00:01.189 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:00:11.209 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:00:21.226 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:00:31.244 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-09T00:00:41.264 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:00:51.278 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:01:01.290 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:01:11.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:01:21.319 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:01:31.336 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:01:41.352 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:01:51.364 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:02:01.379 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:02:11.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:02:21.410 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:02:31.421 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:02:41.434 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:02:51.452 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:03:01.469 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:03:11.487 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:03:21.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:03:31.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:03:41.536 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:03:51.556 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:04:01.573 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:04:11.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:04:21.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:04:31.618 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:04:41.634 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:04:51.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:05:01.662 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:05:11.679 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:05:21.697 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:05:31.715 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:05:41.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-09T00:05:51.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-09T00:06:01.767 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-09T00:06:11.786 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-09T00:06:21.797 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-03-09T00:06:31.812 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:06:41.822 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:06:51.838 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:07:01.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:07:11.865 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:07:21.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:07:31.891 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:07:41.906 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:07:51.923 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:08:01.942 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:08:11.958 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:08:21.968 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:08:31.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:08:41.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:08:52.015 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-03-09T00:09:02.031 DEBUG:teuthology.lock.ops:lock_many request: {'count': 4, 'locked_by': 'scheduled_sage@teuthology', 'description': '/home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152', 'machine_type': 'smithi'} 2020-03-09T00:09:02.050 DEBUG:teuthology.lock.ops:locked ubuntu@smithi148.front.sepia.ceph.com, ubuntu@smithi104.front.sepia.ceph.com, ubuntu@smithi139.front.sepia.ceph.com, ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:09:02.176 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-09T00:09:02.395 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of centos 8.1 2020-03-09T00:09:02.398 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of centos 8.1 2020-03-09T00:09:02.400 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of centos 8.1 2020-03-09T00:09:02.414 INFO:teuthology.provision.fog.smithi050:Scheduling deploy of centos 8.1 2020-03-09T00:09:02.702 INFO:teuthology.orchestra.console:Power off smithi139 2020-03-09T00:09:02.702 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-09T00:09:02.724 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:02.741 INFO:teuthology.orchestra.console:Power off smithi148 2020-03-09T00:09:02.741 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-09T00:09:02.758 INFO:teuthology.orchestra.console:Power off smithi050 2020-03-09T00:09:02.758 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-09T00:09:02.772 INFO:teuthology.orchestra.console:Power off smithi104 2020-03-09T00:09:02.772 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-09T00:09:02.780 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:02.787 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:02.795 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:02.881 INFO:teuthology.orchestra.console:Power off for smithi139 completed 2020-03-09T00:09:02.905 INFO:teuthology.orchestra.console:Power off for smithi050 completed 2020-03-09T00:09:02.916 INFO:teuthology.orchestra.console:Power off for smithi104 completed 2020-03-09T00:09:02.982 INFO:teuthology.orchestra.console:Power on smithi139 2020-03-09T00:09:02.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-09T00:09:03.004 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:03.012 INFO:teuthology.orchestra.console:Power on smithi050 2020-03-09T00:09:03.013 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-09T00:09:03.020 INFO:teuthology.orchestra.console:Power on smithi104 2020-03-09T00:09:03.021 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-09T00:09:03.031 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:03.043 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:06.811 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:07.031 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:07.052 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:07.064 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:07.264 INFO:teuthology.orchestra.console:Power on for smithi050 completed 2020-03-09T00:09:07.365 INFO:teuthology.provision.fog.smithi050:Waiting for deploy to finish 2020-03-09T00:09:10.925 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:11.138 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2020-03-09T00:09:11.143 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:11.186 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:11.239 INFO:teuthology.orchestra.console:Power on smithi148 2020-03-09T00:09:11.240 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-09T00:09:11.262 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:11.353 INFO:teuthology.orchestra.console:Power on for smithi139 completed 2020-03-09T00:09:11.401 INFO:teuthology.orchestra.console:Power on for smithi104 completed 2020-03-09T00:09:11.455 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish 2020-03-09T00:09:11.502 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish 2020-03-09T00:09:15.288 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:19.398 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-09T00:09:19.610 INFO:teuthology.orchestra.console:Power on for smithi148 completed 2020-03-09T00:09:19.711 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2020-03-09T00:11:38.159 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:11:42.273 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:11:47.223 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:11:50.531 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:11:51.351 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:11:56.279 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:11:57.330 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:11:59.607 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:00.406 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:05.335 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:06.391 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:08.663 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:09.462 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:14.392 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:15.451 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:17.718 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:18.518 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:23.447 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:24.503 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:26.775 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:27.574 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:32.503 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:33.560 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:35.833 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:36.630 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:42.619 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:44.887 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:45.686 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:51.671 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:12:53.944 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:13:00.731 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:13:09.786 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:13:17.480 INFO:teuthology.orchestra.run.smithi139:> hostname 2020-03-09T00:13:17.797 INFO:teuthology.orchestra.run.smithi139.stdout:smithi113 2020-03-09T00:13:17.798 INFO:teuthology.orchestra.run.smithi139:> grep smithi113 /etc/hosts 2020-03-09T00:13:17.851 INFO:teuthology.orchestra.run.smithi139.stdout:172.21.15.113 smithi113.front.sepia.ceph.com smithi113 2020-03-09T00:13:17.851 INFO:teuthology.orchestra.run.smithi139:> sudo hostname smithi139 2020-03-09T00:13:18.003 INFO:teuthology.orchestra.run.smithi139:> sudo sed -i -e 's/smithi113/smithi139/g' /etc/hosts 2020-03-09T00:13:18.031 INFO:teuthology.orchestra.run.smithi139:> sudo sed -i -e 's/172.21.15.113/172.21.15.139/g' /etc/hosts 2020-03-09T00:13:18.099 INFO:teuthology.orchestra.run.smithi139:> sudo sed -i -e 's/smithi113/smithi139/g' /etc/hostname 2020-03-09T00:13:18.166 INFO:teuthology.orchestra.run.smithi139:> sudo hostname smithi139 2020-03-09T00:13:18.233 INFO:teuthology.provision.fog.smithi139:Deploy complete! 2020-03-09T00:13:18.235 DEBUG:teuthology.parallel:result is None 2020-03-09T00:13:38.519 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:13:38.961 INFO:teuthology.orchestra.run.smithi050:> hostname 2020-03-09T00:13:39.307 INFO:teuthology.orchestra.run.smithi050.stdout:smithi050.front.sepia.ceph.com 2020-03-09T00:13:39.308 INFO:teuthology.orchestra.run.smithi050:> grep smithi050.front.sepia.ceph.com /etc/hosts 2020-03-09T00:13:39.363 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50 smithi050.front.sepia.ceph.com smithi050 2020-03-09T00:13:39.364 INFO:teuthology.orchestra.run.smithi050:> sudo hostname smithi050 2020-03-09T00:13:39.488 INFO:teuthology.orchestra.run.smithi050:> sudo sed -i -e 's/smithi050.front.sepia.ceph.com/smithi050/g' /etc/hosts 2020-03-09T00:13:39.522 INFO:teuthology.orchestra.run.smithi050:> sudo sed -i -e 's/172.21.15.50/172.21.15.50/g' /etc/hosts 2020-03-09T00:13:39.593 INFO:teuthology.orchestra.run.smithi050:> sudo sed -i -e 's/smithi050.front.sepia.ceph.com/smithi050/g' /etc/hostname 2020-03-09T00:13:39.664 INFO:teuthology.orchestra.run.smithi050:> sudo hostname smithi050 2020-03-09T00:13:39.734 INFO:teuthology.provision.fog.smithi050:Deploy complete! 2020-03-09T00:13:39.735 DEBUG:teuthology.parallel:result is None 2020-03-09T00:13:41.497 INFO:teuthology.orchestra.run.smithi104:> hostname 2020-03-09T00:13:41.828 INFO:teuthology.orchestra.run.smithi104.stdout:smithi113 2020-03-09T00:13:41.829 INFO:teuthology.orchestra.run.smithi104:> grep smithi113 /etc/hosts 2020-03-09T00:13:41.842 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.113 smithi113.front.sepia.ceph.com smithi113 2020-03-09T00:13:41.843 INFO:teuthology.orchestra.run.smithi104:> sudo hostname smithi104 2020-03-09T00:13:42.007 INFO:teuthology.orchestra.run.smithi104:> sudo sed -i -e 's/smithi113/smithi104/g' /etc/hosts 2020-03-09T00:13:42.036 INFO:teuthology.orchestra.run.smithi104:> sudo sed -i -e 's/172.21.15.113/172.21.15.104/g' /etc/hosts 2020-03-09T00:13:42.104 INFO:teuthology.orchestra.run.smithi104:> sudo sed -i -e 's/smithi113/smithi104/g' /etc/hostname 2020-03-09T00:13:42.173 INFO:teuthology.orchestra.run.smithi104:> sudo hostname smithi104 2020-03-09T00:13:42.243 INFO:teuthology.provision.fog.smithi104:Deploy complete! 2020-03-09T00:13:42.244 DEBUG:teuthology.parallel:result is None 2020-03-09T00:13:59.954 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:00.243 INFO:teuthology.orchestra.run.smithi148:> hostname 2020-03-09T00:14:00.562 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com 2020-03-09T00:14:00.562 INFO:teuthology.orchestra.run.smithi148:> grep smithi148.front.sepia.ceph.com /etc/hosts 2020-03-09T00:14:00.577 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148 2020-03-09T00:14:00.577 INFO:teuthology.orchestra.run.smithi148:> sudo hostname smithi148 2020-03-09T00:14:00.685 INFO:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/smithi148.front.sepia.ceph.com/smithi148/g' /etc/hosts 2020-03-09T00:14:00.715 INFO:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/172.21.15.148/172.21.15.148/g' /etc/hosts 2020-03-09T00:14:00.783 INFO:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/smithi148.front.sepia.ceph.com/smithi148/g' /etc/hostname 2020-03-09T00:14:00.852 INFO:teuthology.orchestra.run.smithi148:> sudo hostname smithi148 2020-03-09T00:14:00.919 INFO:teuthology.provision.fog.smithi148:Deploy complete! 2020-03-09T00:14:00.920 DEBUG:teuthology.parallel:result is None 2020-03-09T00:14:01.105 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com 2020-03-09T00:14:01.106 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-09T00:14:01.123 INFO:teuthology.lock.ops:Checking smithi050.front.sepia.ceph.com 2020-03-09T00:14:01.124 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-09T00:14:01.134 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com 2020-03-09T00:14:01.135 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-09T00:14:01.145 INFO:teuthology.lock.ops:Checking smithi139.front.sepia.ceph.com 2020-03-09T00:14:01.145 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-09T00:14:01.159 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server 2020-03-09T00:14:01.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:14:01.160 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:01.229 INFO:teuthology.orchestra.run.smithi148:> true 2020-03-09T00:14:01.388 INFO:teuthology.orchestra.run.smithi148:> uname -m 2020-03-09T00:14:01.441 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2020-03-09T00:14:01.441 INFO:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2020-03-09T00:14:01.495 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Linux" 2020-03-09T00:14:01.495 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8 (Core)" 2020-03-09T00:14:01.495 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2020-03-09T00:14:01.495 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:01.496 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2020-03-09T00:14:01.496 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:01.496 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:01.496 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:01.496 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:01.496 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:01.496 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:01.497 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:01.497 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:01.497 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:01.497 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:01.497 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:01.497 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:01.499 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2020-03-09T00:14:01.517 INFO:teuthology.lock.ops:Updating [smithi050.front.sepia.ceph.com]: set os type and version on server 2020-03-09T00:14:01.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:14:01.518 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:01.588 INFO:teuthology.orchestra.run.smithi050:> true 2020-03-09T00:14:01.772 INFO:teuthology.orchestra.run.smithi050:> uname -m 2020-03-09T00:14:01.827 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2020-03-09T00:14:01.828 INFO:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2020-03-09T00:14:01.882 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Linux" 2020-03-09T00:14:01.882 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8 (Core)" 2020-03-09T00:14:01.882 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2020-03-09T00:14:01.882 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:01.882 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8" 2020-03-09T00:14:01.882 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:01.883 INFO:teuthology.orchestra.run.smithi050.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:01.884 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:01.884 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:01.884 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:01.884 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2020-03-09T00:14:01.897 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server 2020-03-09T00:14:01.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:14:01.898 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:01.974 INFO:teuthology.orchestra.run.smithi104:> true 2020-03-09T00:14:02.140 INFO:teuthology.orchestra.run.smithi104:> uname -m 2020-03-09T00:14:02.194 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2020-03-09T00:14:02.195 INFO:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2020-03-09T00:14:02.248 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Linux" 2020-03-09T00:14:02.248 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8 (Core)" 2020-03-09T00:14:02.248 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2020-03-09T00:14:02.248 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:02.248 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8" 2020-03-09T00:14:02.249 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:02.249 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:02.249 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:02.249 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:02.249 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:02.249 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:02.249 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:02.250 INFO:teuthology.orchestra.run.smithi104.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:02.250 INFO:teuthology.orchestra.run.smithi104.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:02.250 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:02.250 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:02.250 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:02.251 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2020-03-09T00:14:02.266 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: set os type and version on server 2020-03-09T00:14:02.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:14:02.266 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:02.336 INFO:teuthology.orchestra.run.smithi139:> true 2020-03-09T00:14:02.510 INFO:teuthology.orchestra.run.smithi139:> uname -m 2020-03-09T00:14:02.564 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2020-03-09T00:14:02.565 INFO:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2020-03-09T00:14:02.618 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Linux" 2020-03-09T00:14:02.618 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8 (Core)" 2020-03-09T00:14:02.618 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:02.619 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:02.620 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:02.620 INFO:teuthology.orchestra.run.smithi139.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:02.620 INFO:teuthology.orchestra.run.smithi139.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:02.620 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:02.620 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:02.620 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:02.621 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2020-03-09T00:14:02.632 INFO:teuthology.task.internal.lock_machines:4 smithi machines locked this try, 4/4 locked so far 2020-03-09T00:14:02.693 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi050.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDalU+i1g46Gk8/fAPywT5D5W1tLlQMV7TqFtSnJ5N9sdptiKCuJ+hco8RcZUgxZDlurA79xmu952hQiQX4goNPOHDf4nJl+67o/esQFBrAKVHCGzjBrslFUNHYYpghL6wi3/3VEFBAhe5OEC4G8VNTCepruQVlkEb0MAe4g71AaTQmmtijLPIfn+HbBHhuKMmIU2t0yJUYfBVlKBlgbme1FzSF/JLQMgAgeUMz0OQG0EW6REJ/w1Gcjly6FrYCeAeUKV4FH1PP0ADBp4gqvrEuMFDyvjbtHwy3R5EJVbSPEMXJDpZCAO4vJP7fcqsNRzJNQcCg92LrQagC55fEJ/29a2XgkDfbiURtZmJ72bvD906Gcz6TpHFPtgpdRNWJuP9Z3jy2zW2w1ZFiwn7FktTvZS13JLt2wrqJTDYfR1JoqplzwzCtrQKmjnx2NzA2RwfP2hSxx8mtQpiqLpmaFZD77gxSgVL4OQq7aeKVcp0yKJAOsJv++IgRX3Qw6Ob+WS8= smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZGIxAw51Rgx3n37R+EOgbt3eHDwLITnZc3qMfay4K3J+yJpAGW5+X2kD03heXhxobOTq908Boo8CL26wvNDSsJIgOoXhBjiRRJHlRKcebsw6gglsAfAh2njV9CrkcSd5wWYDR5JtPzNAx5c4EDBvRFZm/6VjDU0kksc60UaxBNVgQ92XZLczP3Q92b/XcaEKcsN1Mn+aj3L93pvUZGFdw1JhCLRBtj9H/tKRSVBYpygZIx7TVZlHheqk/SsrtcgoO6kshZfv5Ouc+ZVruv9B17yDruOpWWZlgBeh1gF8fE0l08nzbjlTkm+KVg7RW6pop1U1bj77t2vleKDbCJlL8JpmHWk2R5uBDhGU/RJhN2y0FQJpSaCjCzZORFXGL0DoPqvbV2vh4Eh5v5Ysf5LjmjzPCFCaDfQV1qRtLVzPIkq+dwYy/myQmy9gArGmjDGaHwDqTlCNivCmdCXuby4jhmplipZL29MlsiD/qfksWzcVAaMi0NtKhiL1v5zyBRQM= smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMous9Ne3mtgYoORWx5JJz6ldzqlcHZPKlsZA5t6Hc0q3fjKCFLVGxaCMYjxNQWKVh/i2dgi1ttxZxmco78IZR3vVcFUG6CoWQ+a+o8qvQukOFoMhVOr0wop93wvd/yhf00DIqk/4SPFwlMMpX/LmuXHSMSpwrYzw02DVNTDMaTNtBfsLXSi1BQWxwFh7PZ1VrRix16j/JP8oDbj8BKaA5cf83k1G7d62i5DpaD4eG0pdsFUMnFqZj/m1st54CyV3FlivTgjh0zZtz3rcIi1EZtrXhPW9B9VwtkrBTv4XgEROwr3L7lgXOnX+Gsgu82XIuqLXuungrjZo54QPBbp2WsRPDMGTNAat2FRpWCPcRbic3jmgzakDrA0UptizqjAIh2v4T/htnRft+CqNLiq8AeftpZKcIdVxWCIAxBZpuPhuEOAMqfrna1VIGUPrsbpBz1WlXF075PxlsGuXT3FYvEHwLsks0RyS6xTBEdmWZxmdpyrbEsZdN1FQGXyhn8FM= smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRHf68nfkBAqbAK/zpskKJkcMrbu/r9PJWZKqFOBYfsIjDWlvJ/OqIQF4gLN243MMNZWy+deax+JAh5664fWWO0zcp60EQbjWNAj60/p7ry1CqwJjmS9Y/V7luxfYTtHXLof5/cJNrB+829juP+3OKwhqSuZejN6+0xF/fUitPiPaxRvZPEp2iTYLHqJ0ecJ5RrBcr/xJ+fnjrhwP95AvwukKSz1QwPEbmiZ+BI/wo4oJIybRqckzhzBzBGMVOcOij1xQYudCVRHk0X/45Y127veGbSDQ/IvT5b4Vr1Hagv5MNn4N/SlQbz+EoCCb8gS4Vr7bjLdpsMV5hJvYgN+Zy3J64HrdFSLlDp5uVkWaI0UQge3P80z1dd4VRE86LTfssAUovWEOE1jkixIT609V6GxeSFm7OacLEIEIM+m1cqVDsjsGdIjGFQ5fKHnLoNPYhqQMPKptqc7+DWIRpn9KYw5PO2aGNxGRygvxjFSYAAUhszQ1+TFr/7iBs7/U2CKE= 2020-03-09T00:14:02.693 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-09T00:14:02.880 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-09T00:14:02.888 INFO:teuthology.task.internal:Saving configuration 2020-03-09T00:14:02.907 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-09T00:14:02.916 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-09T00:14:02.931 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-09 00:09:02.037706', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi148.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOdT4to6dfrW/SuCxqCnwUc8/iMljEOz1UlScjRhRl8aNKHk+NbJx0RTxhCDC6PpvIs7xtyl2jfzvOGnLriA+ic=', u'description': u'/home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152'} 2020-03-09T00:14:02.949 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-09 00:09:02.037654', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi050.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZEGcfyMX34I/SaXa4Rn2Ipdp4peRYYy5ScaA56wc4Es2AeRx9UaIrq6NfAR5gLgg0y55PHPnFaH3wIcOPd3PY=', u'description': u'/home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152'} 2020-03-09T00:14:02.963 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-09 00:09:02.037691', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi139.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNuoTwxY6nU6o/RTFXBtsghH8qxPl9a0GxIxsTGOdSVTYoqx6MrZ1hFfHS7xnTShSShmZTbtEHl522eYl5ocIMM=', u'description': u'/home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152'} 2020-03-09T00:14:02.977 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-09 00:09:02.037675', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi104.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLoVx+c79va+NbIgL4+k0KR3G/ITrAs83sxhVkuZc34NM18iW/a02sXwOXzNgdSVXnPKQ9z51C4MjiMhEcm25lc=', u'description': u'/home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152'} 2020-03-09T00:14:02.977 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-09T00:14:02.986 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2020-03-09T00:14:02.987 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2020-03-09T00:14:02.987 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2020-03-09T00:14:02.987 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2020-03-09T00:14:02.987 INFO:teuthology.run_tasks:Running task console_log... 2020-03-09T00:14:03.108 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-09T00:14:03.108 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-09T00:14:03.118 INFO:teuthology.task.internal:Opening connections... 2020-03-09T00:14:03.118 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:14:03.119 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:03.193 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2020-03-09T00:14:03.197 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:03.272 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:14:03.273 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:03.352 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:14:03.353 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:14:03.424 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-09T00:14:03.432 INFO:teuthology.orchestra.run.smithi148:> uname -m 2020-03-09T00:14:03.447 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2020-03-09T00:14:03.447 INFO:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2020-03-09T00:14:03.502 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Linux" 2020-03-09T00:14:03.502 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8 (Core)" 2020-03-09T00:14:03.502 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:03.504 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:03.504 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:03.504 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:03.504 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:03.504 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:03.504 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:03.505 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2020-03-09T00:14:03.743 INFO:teuthology.orchestra.run.smithi104:> uname -m 2020-03-09T00:14:03.759 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2020-03-09T00:14:03.760 INFO:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Linux" 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8 (Core)" 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8" 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:03.815 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:03.817 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:03.817 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2020-03-09T00:14:03.881 INFO:teuthology.orchestra.run.smithi050:> uname -m 2020-03-09T00:14:03.898 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2020-03-09T00:14:03.898 INFO:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2020-03-09T00:14:03.971 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Linux" 2020-03-09T00:14:03.971 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8 (Core)" 2020-03-09T00:14:03.971 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2020-03-09T00:14:03.971 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:03.971 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8" 2020-03-09T00:14:03.971 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:03.972 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:03.972 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:03.972 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:03.972 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:03.972 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:03.972 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:03.972 INFO:teuthology.orchestra.run.smithi050.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:03.973 INFO:teuthology.orchestra.run.smithi050.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:03.973 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:03.973 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:03.973 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:03.973 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2020-03-09T00:14:03.987 INFO:teuthology.orchestra.run.smithi139:> uname -m 2020-03-09T00:14:04.002 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2020-03-09T00:14:04.003 INFO:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2020-03-09T00:14:04.056 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Linux" 2020-03-09T00:14:04.057 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8 (Core)" 2020-03-09T00:14:04.057 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2020-03-09T00:14:04.057 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2020-03-09T00:14:04.057 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8" 2020-03-09T00:14:04.057 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8" 2020-03-09T00:14:04.057 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-09T00:14:04.057 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.centos.org/" 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-09T00:14:04.058 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-09T00:14:04.059 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:04.059 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2020-03-09T00:14:04.069 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-09T00:14:04.091 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-09T00:14:04.100 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-09T00:14:04.101 INFO:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2020-03-09T00:14:04.103 INFO:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2020-03-09T00:14:04.105 INFO:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2020-03-09T00:14:04.107 INFO:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2020-03-09T00:14:04.123 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-09T00:14:04.132 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-09T00:14:04.132 INFO:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2020-03-09T00:14:04.159 INFO:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2020-03-09T00:14:04.161 INFO:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2020-03-09T00:14:04.163 INFO:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2020-03-09T00:14:04.185 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-09T00:14:04.252 INFO:teuthology.run_tasks:Running task kernel... 2020-03-09T00:14:04.279 DEBUG:teuthology.task.kernel:normalized config {'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.11': {'sha1': 'distro', 'kdb': True}, 'osd.10': {'sha1': 'distro', 'kdb': True}, 'osd.13': {'sha1': 'distro', 'kdb': True}, 'osd.12': {'sha1': 'distro', 'kdb': True}, 'osd.15': {'sha1': 'distro', 'kdb': True}, 'osd.14': {'sha1': 'distro', 'kdb': True}, 'osd.9': {'sha1': 'distro', 'kdb': True}, 'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.8': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-09T00:14:04.280 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-09T00:14:04.280 INFO:teuthology.orchestra.run.smithi148:> uname -r 2020-03-09T00:14:04.296 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:04.296 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:04.297 INFO:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2020-03-09T00:14:06.614 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - AppStream 29 kB/s | 4.3 kB 00:00 2020-03-09T00:14:07.210 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - AppStream 11 MB/s | 6.5 MB 00:00 2020-03-09T00:14:08.570 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - Base 17 kB/s | 3.8 kB 00:00 2020-03-09T00:14:10.026 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - Base 3.4 MB/s | 5.0 MB 00:01 2020-03-09T00:14:10.992 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - Extras 7.2 kB/s | 1.5 kB 00:00 2020-03-09T00:14:11.070 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for ceph-el8 owned by ktdreyer 45 kB/s | 3.3 kB 00:00 2020-03-09T00:14:11.291 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for ceph-el8 owned by ktdreyer 172 kB/s | 38 kB 00:00 2020-03-09T00:14:11.405 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 138 kB/s | 5.3 kB 00:00 2020-03-09T00:14:11.726 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 19 MB/s | 6.0 MB 00:00 2020-03-09T00:14:12.994 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 1.1 MB/s | 2.9 kB 00:00 2020-03-09T00:14:12.999 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 6.3 MB/s | 30 kB 00:00 2020-03-09T00:14:13.887 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-09T00:14:13.887 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-09T00:14:14.042 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:14:14.043 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:14:14.043 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:14:14.181 INFO:teuthology.orchestra.run.smithi148:> echo no | sudo yum reinstall kernel || true 2020-03-09T00:14:14.566 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:14:12 AM UTC. 2020-03-09T00:14:14.953 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:14:14.954 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:14:14.954 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:14.954 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:14:14.954 INFO:teuthology.orchestra.run.smithi148.stdout:Reinstalling: 2020-03-09T00:14:14.954 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:14.955 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:14.955 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:14.955 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:14:14.955 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:14:14.955 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:14.962 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.1 M 2020-03-09T00:14:14.962 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 0 2020-03-09T00:14:14.962 INFO:teuthology.orchestra.run.smithi148.stderr:Operation aborted. 2020-03-09T00:14:15.055 INFO:teuthology.orchestra.run.smithi148.stdout:Is this ok [y/N]: 2020-03-09T00:14:15.055 INFO:teuthology.orchestra.run.smithi148:> sudo yum reinstall -y kernel || true 2020-03-09T00:14:15.440 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:03 ago on Mon 09 Mar 2020 12:14:12 AM UTC. 2020-03-09T00:14:15.844 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:14:15.845 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:14:15.845 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:15.845 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:14:15.845 INFO:teuthology.orchestra.run.smithi148.stdout:Reinstalling: 2020-03-09T00:14:15.845 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:15.845 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:15.845 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:15.846 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:14:15.846 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:14:15.846 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:15.846 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.1 M 2020-03-09T00:14:15.847 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 0 2020-03-09T00:14:15.847 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:14:16.786 INFO:teuthology.orchestra.run.smithi148.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 2.0 MB/s | 1.5 MB 00:00 2020-03-09T00:14:16.907 INFO:teuthology.orchestra.run.smithi148.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 1.7 MB/s | 1.5 MB 00:00 2020-03-09T00:14:16.907 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:14:16.908 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2020-03-09T00:14:16.930 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:14:17.102 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:14:17.102 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:14:17.168 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:14:17.168 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:14:17.330 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:14:17.483 INFO:teuthology.orchestra.run.smithi148.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-09T00:14:17.634 INFO:teuthology.orchestra.run.smithi148.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:17.634 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:17.743 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:17.743 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:17.986 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:17.986 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-09T00:14:17.986 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:17.987 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:18.203 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-09T00:14:18.203 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:18.203 INFO:teuthology.orchestra.run.smithi148.stdout:Reinstalled: 2020-03-09T00:14:18.204 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-09T00:14:18.204 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:18.204 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:14:18.341 INFO:teuthology.orchestra.run.smithi148:> rpm -q kernel --last | head -n 1 2020-03-09T00:14:18.452 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-147.el8.x86_64 Mon 09 Mar 2020 12:14:17 AM UTC 2020-03-09T00:14:18.453 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-09T00:14:18.454 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-09T00:14:18.454 INFO:teuthology.orchestra.run.smithi139:> uname -r 2020-03-09T00:14:18.473 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:18.474 INFO:teuthology.task.kernel:Running kernel on smithi139: 4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:18.474 INFO:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2020-03-09T00:14:21.034 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - AppStream 16 kB/s | 4.3 kB 00:00 2020-03-09T00:14:21.845 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - AppStream 8.1 MB/s | 6.5 MB 00:00 2020-03-09T00:14:23.301 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Base 12 kB/s | 3.8 kB 00:00 2020-03-09T00:14:24.442 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Base 4.4 MB/s | 5.0 MB 00:01 2020-03-09T00:14:25.339 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Extras 13 kB/s | 1.5 kB 00:00 2020-03-09T00:14:25.399 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for ceph-el8 owned by ktdreyer 60 kB/s | 3.3 kB 00:00 2020-03-09T00:14:25.501 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for ceph-el8 owned by ktdreyer 375 kB/s | 38 kB 00:00 2020-03-09T00:14:25.623 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 129 kB/s | 5.3 kB 00:00 2020-03-09T00:14:25.931 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 19 MB/s | 6.0 MB 00:00 2020-03-09T00:14:27.148 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 1.3 MB/s | 2.9 kB 00:00 2020-03-09T00:14:27.153 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 6.4 MB/s | 30 kB 00:00 2020-03-09T00:14:28.032 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-09T00:14:28.032 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-09T00:14:28.175 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:14:28.176 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:14:28.176 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:14:28.290 INFO:teuthology.orchestra.run.smithi139:> echo no | sudo yum reinstall kernel || true 2020-03-09T00:14:28.675 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:01 ago on Mon 09 Mar 2020 12:14:27 AM UTC. 2020-03-09T00:14:29.028 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:14:29.028 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:14:29.029 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:29.029 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:14:29.029 INFO:teuthology.orchestra.run.smithi139.stdout:Reinstalling: 2020-03-09T00:14:29.029 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:29.030 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:29.030 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:29.030 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:14:29.030 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:14:29.030 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:29.031 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.1 M 2020-03-09T00:14:29.031 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 0 2020-03-09T00:14:29.031 INFO:teuthology.orchestra.run.smithi139.stderr:Operation aborted. 2020-03-09T00:14:29.121 INFO:teuthology.orchestra.run.smithi139.stdout:Is this ok [y/N]: 2020-03-09T00:14:29.122 INFO:teuthology.orchestra.run.smithi139:> sudo yum reinstall -y kernel || true 2020-03-09T00:14:29.504 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:14:27 AM UTC. 2020-03-09T00:14:29.868 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:14:29.869 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:14:29.869 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:29.870 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:14:29.870 INFO:teuthology.orchestra.run.smithi139.stdout:Reinstalling: 2020-03-09T00:14:29.870 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:29.870 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:29.870 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:29.870 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:14:29.870 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:14:29.871 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:29.871 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.1 M 2020-03-09T00:14:29.872 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 0 2020-03-09T00:14:29.872 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:14:30.813 INFO:teuthology.orchestra.run.smithi139.stdout:(1/2): kernel-4.18.0-147.el8.x86_64.rpm 2.0 MB/s | 1.5 MB 00:00 2020-03-09T00:14:30.817 INFO:teuthology.orchestra.run.smithi139.stdout:(2/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 2.0 MB/s | 1.5 MB 00:00 2020-03-09T00:14:30.818 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:14:30.818 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.2 MB/s | 3.1 MB 00:00 2020-03-09T00:14:30.845 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:14:31.014 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:14:31.015 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:14:31.104 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:14:31.104 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:14:31.239 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:14:31.396 INFO:teuthology.orchestra.run.smithi139.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-09T00:14:31.505 INFO:teuthology.orchestra.run.smithi139.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:31.505 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:31.605 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:31.606 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:31.830 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:31.830 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-09T00:14:31.830 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:31.830 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:32.059 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-09T00:14:32.060 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:32.060 INFO:teuthology.orchestra.run.smithi139.stdout:Reinstalled: 2020-03-09T00:14:32.060 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-09T00:14:32.060 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:14:32.060 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:14:32.150 INFO:teuthology.orchestra.run.smithi139:> rpm -q kernel --last | head -n 1 2020-03-09T00:14:32.208 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-147.el8.x86_64 Mon 09 Mar 2020 12:14:31 AM UTC 2020-03-09T00:14:32.209 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-09T00:14:32.209 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-09T00:14:32.210 INFO:teuthology.orchestra.run.smithi050:> uname -r 2020-03-09T00:14:32.227 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:32.228 INFO:teuthology.task.kernel:Running kernel on smithi050: 4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:32.228 INFO:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2020-03-09T00:14:34.986 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - AppStream 14 kB/s | 4.3 kB 00:00 2020-03-09T00:14:35.731 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - AppStream 8.8 MB/s | 6.5 MB 00:00 2020-03-09T00:14:37.086 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - Base 29 kB/s | 3.8 kB 00:00 2020-03-09T00:14:37.537 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - Base 11 MB/s | 5.0 MB 00:00 2020-03-09T00:14:38.426 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - Extras 19 kB/s | 1.5 kB 00:00 2020-03-09T00:14:38.502 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for ceph-el8 owned by ktdreyer 51 kB/s | 3.3 kB 00:00 2020-03-09T00:14:38.602 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for ceph-el8 owned by ktdreyer 382 kB/s | 38 kB 00:00 2020-03-09T00:14:38.731 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 151 kB/s | 5.3 kB 00:00 2020-03-09T00:14:39.089 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 17 MB/s | 6.0 MB 00:00 2020-03-09T00:14:40.437 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 1.3 MB/s | 2.9 kB 00:00 2020-03-09T00:14:40.442 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 6.0 MB/s | 30 kB 00:00 2020-03-09T00:14:41.383 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-09T00:14:41.383 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-09T00:14:41.544 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:14:41.545 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:14:41.546 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:14:41.702 INFO:teuthology.orchestra.run.smithi050:> echo no | sudo yum reinstall kernel || true 2020-03-09T00:14:42.117 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:14:40 AM UTC. 2020-03-09T00:14:42.505 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:14:42.505 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:14:42.505 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:42.505 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:14:42.506 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2020-03-09T00:14:42.506 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:42.506 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:42.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:42.506 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:14:42.506 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:14:42.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:42.508 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.1 M 2020-03-09T00:14:42.508 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2020-03-09T00:14:42.508 INFO:teuthology.orchestra.run.smithi050.stderr:Operation aborted. 2020-03-09T00:14:42.620 INFO:teuthology.orchestra.run.smithi050.stdout:Is this ok [y/N]: 2020-03-09T00:14:42.620 INFO:teuthology.orchestra.run.smithi050:> sudo yum reinstall -y kernel || true 2020-03-09T00:14:43.034 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Mon 09 Mar 2020 12:14:40 AM UTC. 2020-03-09T00:14:43.450 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:14:43.450 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:14:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalling: 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:14:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:43.452 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.1 M 2020-03-09T00:14:43.452 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 0 2020-03-09T00:14:43.452 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:14:44.381 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 2.0 MB/s | 1.5 MB 00:00 2020-03-09T00:14:44.433 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 1.9 MB/s | 1.5 MB 00:00 2020-03-09T00:14:44.433 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:14:44.434 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.1 MB/s | 3.1 MB 00:00 2020-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:14:44.661 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:14:44.661 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:14:44.746 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:14:44.746 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:14:44.890 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:14:45.044 INFO:teuthology.orchestra.run.smithi050.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-09T00:14:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:45.187 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-09T00:14:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:45.740 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-09T00:14:45.740 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:45.740 INFO:teuthology.orchestra.run.smithi050.stdout:Reinstalled: 2020-03-09T00:14:45.740 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-09T00:14:45.740 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:14:45.740 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:14:45.851 INFO:teuthology.orchestra.run.smithi050:> rpm -q kernel --last | head -n 1 2020-03-09T00:14:45.971 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-147.5.1.el8_1.x86_64 Mon 09 Mar 2020 12:14:45 AM UTC 2020-03-09T00:14:45.972 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:45.972 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-03-09T00:14:45.972 INFO:teuthology.orchestra.run.smithi104:> uname -r 2020-03-09T00:14:45.989 INFO:teuthology.orchestra.run.smithi104.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:45.990 INFO:teuthology.task.kernel:Running kernel on smithi104: 4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:45.990 INFO:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2020-03-09T00:14:48.723 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - AppStream 14 kB/s | 4.3 kB 00:00 2020-03-09T00:14:49.519 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - AppStream 8.2 MB/s | 6.5 MB 00:00 2020-03-09T00:14:50.935 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - Base 13 kB/s | 3.8 kB 00:00 2020-03-09T00:14:51.483 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - Base 9.1 MB/s | 5.0 MB 00:00 2020-03-09T00:14:52.502 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - Extras 6.0 kB/s | 1.5 kB 00:00 2020-03-09T00:14:52.577 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for ceph-el8 owned by ktdreyer 53 kB/s | 3.3 kB 00:00 2020-03-09T00:14:52.675 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for ceph-el8 owned by ktdreyer 385 kB/s | 38 kB 00:00 2020-03-09T00:14:52.785 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 115 kB/s | 5.3 kB 00:00 2020-03-09T00:14:53.124 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 18 MB/s | 6.0 MB 00:00 2020-03-09T00:14:54.352 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 1.2 MB/s | 2.9 kB 00:00 2020-03-09T00:14:54.357 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 5.9 MB/s | 30 kB 00:00 2020-03-09T00:14:55.231 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-09T00:14:55.231 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-09T00:14:55.384 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:14:55.385 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:14:55.386 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:14:55.525 INFO:teuthology.orchestra.run.smithi104:> echo no | sudo yum reinstall kernel || true 2020-03-09T00:14:55.951 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:01 ago on Mon 09 Mar 2020 12:14:54 AM UTC. 2020-03-09T00:14:56.301 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:14:56.302 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:14:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:56.302 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:14:56.302 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalling: 2020-03-09T00:14:56.302 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:56.303 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:14:56.303 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:14:56.303 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:56.304 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.1 M 2020-03-09T00:14:56.304 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 0 2020-03-09T00:14:56.304 INFO:teuthology.orchestra.run.smithi104.stderr:Operation aborted. 2020-03-09T00:14:56.394 INFO:teuthology.orchestra.run.smithi104.stdout:Is this ok [y/N]: 2020-03-09T00:14:56.394 INFO:teuthology.orchestra.run.smithi104:> sudo yum reinstall -y kernel || true 2020-03-09T00:14:56.776 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:14:54 AM UTC. 2020-03-09T00:14:57.135 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:14:57.136 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:14:57.136 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2020-03-09T00:14:57.136 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:14:57.136 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalling: 2020-03-09T00:14:57.136 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 BaseOS 1.5 M 2020-03-09T00:14:57.136 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 4.18.0-147.el8 BaseOS 1.5 M 2020-03-09T00:14:57.137 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:57.137 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:14:57.137 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:14:57.137 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:57.137 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.1 M 2020-03-09T00:14:57.137 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 0 2020-03-09T00:14:57.137 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:14:57.521 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 5.2 MB/s | 1.5 MB 00:00 2020-03-09T00:14:57.547 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 4.8 MB/s | 1.5 MB 00:00 2020-03-09T00:14:57.547 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:14:57.547 INFO:teuthology.orchestra.run.smithi104.stdout:Total 7.4 MB/s | 3.1 MB 00:00 2020-03-09T00:14:57.570 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:14:57.730 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:14:57.731 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:14:57.802 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:14:57.802 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:14:57.939 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:14:58.102 INFO:teuthology.orchestra.run.smithi104.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-09T00:14:58.251 INFO:teuthology.orchestra.run.smithi104.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:58.251 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:58.354 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:58.355 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:58.578 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-09T00:14:58.579 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-09T00:14:58.579 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-09T00:14:58.579 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-09T00:14:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-09T00:14:58.796 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:58.796 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalled: 2020-03-09T00:14:58.797 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-09T00:14:58.797 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:14:58.797 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:14:58.902 INFO:teuthology.orchestra.run.smithi104:> rpm -q kernel --last | head -n 1 2020-03-09T00:14:58.954 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-4.18.0-147.5.1.el8_1.x86_64 Mon 09 Mar 2020 12:14:58 AM UTC 2020-03-09T00:14:58.956 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.5.1.el8_1.x86_64 2020-03-09T00:14:58.956 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-03-09T00:14:58.956 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-09T00:14:58.956 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-09T00:14:58.956 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-09T00:14:58.956 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-09T00:14:58.957 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2020-03-09T00:14:58.957 INFO:teuthology.orchestra.run.smithi148:> rpm -q kernel --last | head -n 1 2020-03-09T00:14:59.005 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-147.el8.x86_64 Mon 09 Mar 2020 12:14:17 AM UTC 2020-03-09T00:14:59.006 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-09T00:14:59.006 INFO:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2020-03-09T00:14:59.040 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.02 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 78.el8 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Tue 11 Feb 2020 03:32:52 PM UTC 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:Group : System Environment/Base 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 9192691 2020-03-09T00:14:59.041 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2020-03-09T00:14:59.042 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-09T00:14:59.042 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-09T00:14:59.042 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-09T00:14:59.042 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-09T00:14:59.042 INFO:teuthology.orchestra.run.smithi148.stdout:Relocations : (not relocatable) 2020-03-09T00:14:59.042 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : CentOS BuildSystem 2020-03-09T00:14:59.042 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-09T00:14:59.043 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2020-03-09T00:14:59.044 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:14:59.044 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2020-03-09T00:14:59.044 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-09T00:14:59.045 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 4.18.0-147.el8.x86_64 2020-03-09T00:14:59.045 INFO:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-09T00:14:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2020-03-09T00:15:02.107 INFO:teuthology.orchestra.run.smithi148.stderr:done 2020-03-09T00:15:02.108 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:15:02.139 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.D9sbnVjjGd 2020-03-09T00:15:02.140 INFO:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.D9sbnVjjGd 2020-03-09T00:15:02.218 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.D9sbnVjjGd 2020-03-09T00:15:02.377 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.D9sbnVjjGd is 4KB 2020-03-09T00:15:02.426 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.D9sbnVjjGd 2020-03-09T00:15:02.442 INFO:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2020-03-09T00:15:02.513 INFO:teuthology.orchestra.run.smithi148.stdout:64c9091786944d63a01cf35d7b3c4063-0-rescue.conf 2020-03-09T00:15:02.513 INFO:teuthology.orchestra.run.smithi148.stdout:64c9091786944d63a01cf35d7b3c4063-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-09T00:15:02.514 INFO:teuthology.orchestra.run.smithi148.stdout:64c9091786944d63a01cf35d7b3c4063-4.18.0-147.el8.x86_64.conf 2020-03-09T00:15:02.515 INFO:teuthology.orchestra.run.smithi148:> sudo grub2-set-default 64c9091786944d63a01cf35d7b3c4063-4.18.0-147.el8.x86_64 2020-03-09T00:15:02.687 INFO:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2020-03-09T00:15:02.689 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2020-03-09T00:15:02.689 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2020-03-09T00:15:02.689 INFO:teuthology.orchestra.run.smithi139:> rpm -q kernel --last | head -n 1 2020-03-09T00:15:02.736 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-147.el8.x86_64 Mon 09 Mar 2020 12:14:31 AM UTC 2020-03-09T00:15:02.738 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-09T00:15:02.739 INFO:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2020-03-09T00:15:02.774 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2020-03-09T00:15:02.774 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2020-03-09T00:15:02.774 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.02 2020-03-09T00:15:02.774 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 78.el8 2020-03-09T00:15:02.775 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2020-03-09T00:15:02.775 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Tue 11 Feb 2020 03:32:52 PM UTC 2020-03-09T00:15:02.775 INFO:teuthology.orchestra.run.smithi139.stdout:Group : System Environment/Base 2020-03-09T00:15:02.775 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 9192691 2020-03-09T00:15:02.775 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2020-03-09T00:15:02.776 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-09T00:15:02.776 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-09T00:15:02.776 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-09T00:15:02.776 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-09T00:15:02.777 INFO:teuthology.orchestra.run.smithi139.stdout:Relocations : (not relocatable) 2020-03-09T00:15:02.777 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : CentOS BuildSystem 2020-03-09T00:15:02.777 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2020-03-09T00:15:02.777 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-09T00:15:02.778 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2020-03-09T00:15:02.778 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2020-03-09T00:15:02.778 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:15:02.778 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-09T00:15:02.778 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-09T00:15:02.779 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-09T00:15:02.779 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2020-03-09T00:15:02.779 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:15:02.779 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2020-03-09T00:15:02.781 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-09T00:15:02.781 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 4.18.0-147.el8.x86_64 2020-03-09T00:15:02.781 INFO:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-09T00:15:03.567 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2020-03-09T00:15:05.381 INFO:teuthology.orchestra.run.smithi139.stderr:done 2020-03-09T00:15:05.382 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:15:05.404 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.sHXibgPj70 2020-03-09T00:15:05.405 INFO:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sHXibgPj70 2020-03-09T00:15:05.486 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.sHXibgPj70 2020-03-09T00:15:05.638 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.sHXibgPj70 is 4KB 2020-03-09T00:15:05.686 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.sHXibgPj70 2020-03-09T00:15:05.699 INFO:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2020-03-09T00:15:05.764 INFO:teuthology.orchestra.run.smithi139.stdout:64c9091786944d63a01cf35d7b3c4063-0-rescue.conf 2020-03-09T00:15:05.765 INFO:teuthology.orchestra.run.smithi139.stdout:64c9091786944d63a01cf35d7b3c4063-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-09T00:15:05.765 INFO:teuthology.orchestra.run.smithi139.stdout:64c9091786944d63a01cf35d7b3c4063-4.18.0-147.el8.x86_64.conf 2020-03-09T00:15:05.766 INFO:teuthology.orchestra.run.smithi139:> sudo grub2-set-default 64c9091786944d63a01cf35d7b3c4063-4.18.0-147.el8.x86_64 2020-03-09T00:15:05.907 INFO:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2020-03-09T00:15:35.910 INFO:teuthology.misc:Re-opening connections... 2020-03-09T00:15:35.910 INFO:teuthology.misc:trying to connect to ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:15:35.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:15:35.911 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:15:54.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2020-03-09T00:15:54.342 INFO:teuthology.misc:trying to connect to ubuntu@smithi104.front.sepia.ceph.com 2020-03-09T00:15:54.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:15:54.344 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:15:54.417 INFO:teuthology.orchestra.run.smithi104:> true 2020-03-09T00:15:54.579 INFO:teuthology.misc:trying to connect to ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:15:54.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:15:54.580 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:16:54.581 DEBUG:teuthology.orchestra.remote:timed out 2020-03-09T00:16:54.582 DEBUG:teuthology.misc:waited 78.6717970371 2020-03-09T00:16:55.582 INFO:teuthology.misc:trying to connect to ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:16:55.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:16:55.583 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:16:55.866 INFO:teuthology.orchestra.run.smithi148:> true 2020-03-09T00:16:56.175 INFO:teuthology.misc:trying to connect to ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:16:56.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:16:56.176 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:16:56.575 INFO:teuthology.orchestra.run.smithi139:> true 2020-03-09T00:16:56.871 DEBUG:teuthology.misc:waited 80.9611439705 2020-03-09T00:16:57.872 INFO:teuthology.misc:trying to connect to ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:16:57.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:16:57.874 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:16:57.948 INFO:teuthology.orchestra.run.smithi050:> true 2020-03-09T00:16:58.128 DEBUG:teuthology.misc:waited 82.2182359695 2020-03-09T00:16:59.129 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-09T00:16:59.129 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-09T00:16:59.129 INFO:teuthology.orchestra.run.smithi148:> uname -r 2020-03-09T00:16:59.147 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-147.el8.x86_64 2020-03-09T00:16:59.147 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-09T00:16:59.147 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-09T00:16:59.147 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2020-03-09T00:16:59.147 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-147.el8.x86_64"... 2020-03-09T00:16:59.148 INFO:teuthology.orchestra.run.smithi139:> uname -r 2020-03-09T00:16:59.165 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-147.el8.x86_64 2020-03-09T00:16:59.165 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-09T00:16:59.165 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-09T00:17:00.166 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-09T00:17:00.167 INFO:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-09T00:17:00.252 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2020-03-09T00:17:00.267 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2020-03-09T00:17:00.267 INFO:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-09T00:17:00.355 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2020-03-09T00:17:00.370 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2020-03-09T00:17:00.372 INFO:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-09T00:17:00.418 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2020-03-09T00:17:00.433 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2020-03-09T00:17:00.434 INFO:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-09T00:17:00.480 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2020-03-09T00:17:00.495 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-09T00:17:00.504 INFO:teuthology.task.internal:Creating test directory... 2020-03-09T00:17:00.505 INFO:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-09T00:17:00.507 INFO:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-09T00:17:00.510 INFO:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-09T00:17:00.513 INFO:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-09T00:17:00.530 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-09T00:17:00.540 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-09T00:17:00.566 INFO:teuthology.task.internal:Creating archive directory... 2020-03-09T00:17:00.566 INFO:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-09T00:17:00.573 INFO:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-09T00:17:00.575 INFO:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-09T00:17:00.578 INFO:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-09T00:17:00.600 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-09T00:17:00.613 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-09T00:17:00.614 INFO:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-09T00:17:00.634 INFO:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-09T00:17:00.638 INFO:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-09T00:17:00.640 INFO:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-09T00:17:00.673 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.674 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.676 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.688 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.688 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.695 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.699 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.714 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-09T00:17:00.715 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-09T00:17:00.723 INFO:teuthology.task.internal:Configuring sudo... 2020-03-09T00:17:00.724 INFO:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-09T00:17:00.738 INFO:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-09T00:17:00.741 INFO:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-09T00:17:00.743 INFO:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-09T00:17:00.772 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-09T00:17:00.783 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-09T00:17:00.783 INFO:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-09T00:17:00.813 INFO:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-09T00:17:00.815 INFO:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-09T00:17:00.817 INFO:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-09T00:17:00.832 INFO:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:00.905 INFO:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:00.973 INFO:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:01.044 INFO:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:01.096 INFO:teuthology.orchestra.run.smithi148:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-09T00:17:01.181 INFO:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:01.214 INFO:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:01.277 INFO:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:01.350 INFO:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:01.403 INFO:teuthology.orchestra.run.smithi104:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-09T00:17:01.488 INFO:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:01.523 INFO:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:01.598 INFO:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:01.632 INFO:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:01.690 INFO:teuthology.orchestra.run.smithi050:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-09T00:17:01.775 INFO:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:01.809 INFO:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-09T00:17:01.877 INFO:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:01.952 INFO:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-09T00:17:02.007 INFO:teuthology.orchestra.run.smithi139:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-09T00:17:02.092 INFO:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2020-03-09T00:17:02.094 INFO:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2020-03-09T00:17:02.097 INFO:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2020-03-09T00:17:02.135 INFO:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2020-03-09T00:17:02.192 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-09T00:17:02.201 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-09T00:17:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-09T00:17:02.228 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-09T00:17:02.675 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-09T00:17:02.683 INFO:teuthology.task.internal:Starting timer... 2020-03-09T00:17:02.683 INFO:teuthology.run_tasks:Running task pcp... 2020-03-09T00:17:02.714 INFO:teuthology.run_tasks:Running task selinux... 2020-03-09T00:17:02.724 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-09T00:17:02.724 INFO:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2020-03-09T00:17:02.796 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: [ OK ] 2020-03-09T00:17:02.798 INFO:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2020-03-09T00:17:02.861 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: [ OK ] 2020-03-09T00:17:02.862 INFO:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2020-03-09T00:17:02.938 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: [ OK ] 2020-03-09T00:17:02.940 INFO:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2020-03-09T00:17:03.024 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: [ OK ] 2020-03-09T00:17:03.025 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-09T00:17:03.026 INFO:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2020-03-09T00:17:03.046 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2020-03-09T00:17:03.047 INFO:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2020-03-09T00:17:03.076 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2020-03-09T00:17:03.077 INFO:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2020-03-09T00:17:03.101 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2020-03-09T00:17:03.101 INFO:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2020-03-09T00:17:03.122 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2020-03-09T00:17:03.123 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2020-03-09T00:17:03.123 INFO:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-09T00:17:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-09T00:17:03.151 INFO:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-09T00:17:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-09T00:17:03.178 INFO:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-09T00:17:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-09T00:17:03.212 INFO:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-09T00:17:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-09T00:17:03.240 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-09T00:17:03.241 INFO:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2020-03-09T00:17:03.268 INFO:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2020-03-09T00:17:03.301 INFO:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2020-03-09T00:17:03.338 INFO:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2020-03-09T00:17:03.366 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-09T00:17:03.376 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-03-09T00:17:03.383 INFO:teuthology.repo_utils:Fetching master from origin 2020-03-09T00:17:03.434 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-09T00:17:03.449 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-09T00:17:03.451 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com,smithi104.front.sepia.ceph.com,smithi050.front.sepia.ceph.com,smithi139.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-09T00:17:03.804 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-09T00:17:05.079 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-09T00:17:08.066 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-09T00:17:08.090 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:08 +0000 (0:00:00.063) 0:00:00.063 ********** 2020-03-09T00:17:08.103 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:08 +0000 (0:00:00.013) 0:00:00.076 ********** 2020-03-09T00:17:08.116 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:08 +0000 (0:00:00.012) 0:00:00.088 ********** 2020-03-09T00:17:08.127 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:08 +0000 (0:00:00.010) 0:00:00.099 ********** 2020-03-09T00:17:10.609 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-03-09T00:17:10.610 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] 2020-03-09T00:17:10.669 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] 2020-03-09T00:17:10.679 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2020-03-09T00:17:10.690 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] 2020-03-09T00:17:10.754 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:10 +0000 (0:00:02.627) 0:00:02.727 ********** 2020-03-09T00:17:10.844 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:10 +0000 (0:00:00.089) 0:00:02.816 ********** 2020-03-09T00:17:10.875 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:10 +0000 (0:00:00.031) 0:00:02.847 ********** 2020-03-09T00:17:10.906 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:10 +0000 (0:00:00.031) 0:00:02.879 ********** 2020-03-09T00:17:11.079 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-09T00:17:11.081 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-09T00:17:11.083 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-09T00:17:11.137 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.231) 0:00:03.110 ********** 2020-03-09T00:17:11.170 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.032) 0:00:03.143 ********** 2020-03-09T00:17:11.175 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-09T00:17:11.176 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-09T00:17:11.213 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.042) 0:00:03.186 ********** 2020-03-09T00:17:11.246 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.033) 0:00:03.219 ********** 2020-03-09T00:17:11.705 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-09T00:17:11.707 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-09T00:17:11.709 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-09T00:17:11.746 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.499) 0:00:03.719 ********** 2020-03-09T00:17:11.775 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.028) 0:00:03.748 ********** 2020-03-09T00:17:11.780 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-09T00:17:11.782 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-09T00:17:11.783 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-09T00:17:11.783 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:11.817 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.042) 0:00:03.790 ********** 2020-03-09T00:17:11.848 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.030) 0:00:03.821 ********** 2020-03-09T00:17:11.878 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.029) 0:00:03.850 ********** 2020-03-09T00:17:11.883 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:11.883 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:11.919 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.041) 0:00:03.892 ********** 2020-03-09T00:17:11.950 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.030) 0:00:03.922 ********** 2020-03-09T00:17:11.955 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:11.994 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:11 +0000 (0:00:00.044) 0:00:03.966 ********** 2020-03-09T00:17:12.176 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-09T00:17:12.176 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:12.178 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:12.179 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:12.215 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:12 +0000 (0:00:00.220) 0:00:04.187 ********** 2020-03-09T00:17:12.245 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:12 +0000 (0:00:00.030) 0:00:04.218 ********** 2020-03-09T00:17:12.275 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:12 +0000 (0:00:00.029) 0:00:04.248 ********** 2020-03-09T00:17:12.281 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:12.319 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:12 +0000 (0:00:00.043) 0:00:04.292 ********** 2020-03-09T00:17:13.150 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-09T00:17:13.150 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713032.29-140723128540025/source", "state": "file", "uid": 0} 2020-03-09T00:17:13.152 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713032.26-129731947034100/source", "state": "file", "uid": 0} 2020-03-09T00:17:13.190 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:13 +0000 (0:00:00.870) 0:00:05.163 ********** 2020-03-09T00:17:13.222 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:13 +0000 (0:00:00.030) 0:00:05.194 ********** 2020-03-09T00:17:13.227 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713032.23-244643226549940/source", "state": "file", "uid": 0} 2020-03-09T00:17:13.228 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713032.33-45719901988251/source", "state": "file", "uid": 0} 2020-03-09T00:17:13.266 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:13 +0000 (0:00:00.044) 0:00:05.238 ********** 2020-03-09T00:17:13.297 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:13 +0000 (0:00:00.031) 0:00:05.270 ********** 2020-03-09T00:17:13.698 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-09T00:17:13.699 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-09T00:17:13.700 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-09T00:17:13.742 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-09T00:17:13.743 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2020-03-09T00:17:13.745 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2020-03-09T00:17:13.746 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEz 2020-03-09T00:17:13.748 INFO:teuthology.task.ansible.out:IKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-09T00:17:13.954 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-09T00:17:13.960 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-09T00:17:14.146 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:14 +0000 (0:00:00.848) 0:00:06.119 ********** 2020-03-09T00:17:14.179 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:14 +0000 (0:00:00.032) 0:00:06.152 ********** 2020-03-09T00:17:14.211 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:14 +0000 (0:00:00.031) 0:00:06.183 ********** 2020-03-09T00:17:14.244 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:14 +0000 (0:00:00.033) 0:00:06.216 ********** 2020-03-09T00:17:14.828 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi050.front.sepia.ceph.com] 2020-03-09T00:17:14.839 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] 2020-03-09T00:17:14.910 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2020-03-09T00:17:14.922 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] 2020-03-09T00:17:14.966 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-09T00:17:14.968 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:14 +0000 (0:00:00.722) 0:00:06.939 ********** 2020-03-09T00:17:15.007 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.039) 0:00:06.979 ********** 2020-03-09T00:17:15.109 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.102) 0:00:07.081 ********** 2020-03-09T00:17:15.147 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.038) 0:00:07.119 ********** 2020-03-09T00:17:15.610 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-09T00:17:15.611 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.222480", "end": "2020-03-09 00:17:15.584946", "rc": 0, "start": "2020-03-09 00:17:15.362466", "stderr": "", "stderr_lines": [], "stdout": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "stdout_lines": ["d9f2fa52162dd365bfcfaae13ab61fb6caff6e16"]} 2020-03-09T00:17:15.659 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.511) 0:00:07.631 ********** 2020-03-09T00:17:15.666 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.234333", "end": "2020-03-09 00:17:15.610209", "rc": 0, "start": "2020-03-09 00:17:15.375876", "stderr": "", "stderr_lines": [], "stdout": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "stdout_lines": ["d9f2fa52162dd365bfcfaae13ab61fb6caff6e16"]} 2020-03-09T00:17:15.667 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.242987", "end": "2020-03-09 00:17:15.616309", "rc": 0, "start": "2020-03-09 00:17:15.373322", "stderr": "", "stderr_lines": [], "stdout": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "stdout_lines": ["d9f2fa52162dd365bfcfaae13ab61fb6caff6e16"]} 2020-03-09T00:17:15.669 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.251158", "end": "2020-03-09 00:17:15.613301", "rc": 0, "start": "2020-03-09 00:17:15.362143", "stderr": "", "stderr_lines": [], "stdout": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "stdout_lines": ["d9f2fa52162dd365bfcfaae13ab61fb6caff6e16"]} 2020-03-09T00:17:15.712 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.053) 0:00:07.685 ********** 2020-03-09T00:17:15.748 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.035) 0:00:07.720 ********** 2020-03-09T00:17:15.782 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.034) 0:00:07.755 ********** 2020-03-09T00:17:15.891 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-09T00:17:15.892 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013072", "end": "2020-03-09 00:17:15.868853", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:17:15.855781", "stderr": "", "stderr_lines": [], "stdout": "966e9a6bfb22958c64d4123b47ddee87a2318d5a", "stdout_lines": ["966e9a6bfb22958c64d4123b47ddee87a2318d5a"]} 2020-03-09T00:17:15.937 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.155) 0:00:07.910 ********** 2020-03-09T00:17:15.945 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008911", "end": "2020-03-09 00:17:15.898478", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:17:15.889567", "stderr": "", "stderr_lines": [], "stdout": "966e9a6bfb22958c64d4123b47ddee87a2318d5a", "stdout_lines": ["966e9a6bfb22958c64d4123b47ddee87a2318d5a"]} 2020-03-09T00:17:15.987 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:15 +0000 (0:00:00.049) 0:00:07.959 ********** 2020-03-09T00:17:15.991 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-09T00:17:15.993 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-09T00:17:15.994 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.017536", "end": "2020-03-09 00:17:15.945938", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:17:15.928402", "stderr": "", "stderr_lines": [], "stdout": "966e9a6bfb22958c64d4123b47ddee87a2318d5a", "stdout_lines": ["966e9a6bfb22958c64d4123b47ddee87a2318d5a"]} 2020-03-09T00:17:16.032 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.044) 0:00:08.004 ********** 2020-03-09T00:17:16.066 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.034) 0:00:08.039 ********** 2020-03-09T00:17:16.071 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011185", "end": "2020-03-09 00:17:15.985623", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:17:15.974438", "stderr": "", "stderr_lines": [], "stdout": "966e9a6bfb22958c64d4123b47ddee87a2318d5a", "stdout_lines": ["966e9a6bfb22958c64d4123b47ddee87a2318d5a"]} 2020-03-09T00:17:16.072 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.074 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-09T00:17:16.075 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.110 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.043) 0:00:08.082 ********** 2020-03-09T00:17:16.142 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.031) 0:00:08.114 ********** 2020-03-09T00:17:16.175 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.032) 0:00:08.147 ********** 2020-03-09T00:17:16.179 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-09T00:17:16.180 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.182 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.222 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.047) 0:00:08.195 ********** 2020-03-09T00:17:16.259 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.036) 0:00:08.231 ********** 2020-03-09T00:17:16.296 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.036) 0:00:08.268 ********** 2020-03-09T00:17:16.302 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-09T00:17:16.303 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Filter the managed_users list] *********************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.303 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-09T00:17:16.304 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.343 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.047) 0:00:08.316 ********** 2020-03-09T00:17:16.376 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.032) 0:00:08.349 ********** 2020-03-09T00:17:16.410 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.033) 0:00:08.382 ********** 2020-03-09T00:17:16.415 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-09T00:17:16.416 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.417 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.455 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.045) 0:00:08.427 ********** 2020-03-09T00:17:16.489 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.033) 0:00:08.461 ********** 2020-03-09T00:17:16.530 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.040) 0:00:08.502 ********** 2020-03-09T00:17:16.536 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-09T00:17:16.538 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.582 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.052) 0:00:08.554 ********** 2020-03-09T00:17:16.615 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.032) 0:00:08.587 ********** 2020-03-09T00:17:16.620 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-09T00:17:16.620 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.659 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.044) 0:00:08.632 ********** 2020-03-09T00:17:16.665 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.709 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.049) 0:00:08.681 ********** 2020-03-09T00:17:16.715 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-09T00:17:16.716 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:16.717 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.755 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.045) 0:00:08.727 ********** 2020-03-09T00:17:16.789 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.034) 0:00:08.761 ********** 2020-03-09T00:17:16.794 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.795 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-09T00:17:16.796 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:16.834 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.045) 0:00:08.807 ********** 2020-03-09T00:17:16.869 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.034) 0:00:08.841 ********** 2020-03-09T00:17:16.874 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-09T00:17:16.876 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.913 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.044) 0:00:08.886 ********** 2020-03-09T00:17:16.919 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:16.956 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:16 +0000 (0:00:00.042) 0:00:08.929 ********** 2020-03-09T00:17:17.003 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-09T00:17:17.004 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-03-09T00:17:17.078 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-03-09T00:17:17.121 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-09T00:17:17.123 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:17.158 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:17 +0000 (0:00:00.201) 0:00:09.130 ********** 2020-03-09T00:17:17.165 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:17:17.229 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:17 +0000 (0:00:00.071) 0:00:09.202 ********** 2020-03-09T00:17:17.235 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-09T00:17:17.235 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-03-09T00:17:17.279 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-03-09T00:17:17.410 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-03-09T00:17:17.422 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-03-09T00:17:17.481 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-03-09T00:17:17.482 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-03-09T00:17:17.611 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-03-09T00:17:17.634 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-03-09T00:17:17.657 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-03-09T00:17:17.679 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-03-09T00:17:17.826 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-03-09T00:17:17.838 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-03-09T00:17:17.861 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-03-09T00:17:17.872 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-03-09T00:17:18.041 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-03-09T00:17:18.063 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-03-09T00:17:18.064 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-03-09T00:17:18.066 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-03-09T00:17:18.255 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-03-09T00:17:18.267 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-03-09T00:17:18.268 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-03-09T00:17:18.291 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-03-09T00:17:18.460 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-03-09T00:17:18.461 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-03-09T00:17:18.462 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-03-09T00:17:18.506 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-03-09T00:17:18.654 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-03-09T00:17:18.666 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-03-09T00:17:18.667 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-03-09T00:17:18.721 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-03-09T00:17:18.858 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-03-09T00:17:18.870 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-03-09T00:17:18.882 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-03-09T00:17:18.936 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-03-09T00:17:19.062 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-03-09T00:17:19.074 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-03-09T00:17:19.076 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-03-09T00:17:19.154 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-03-09T00:17:19.269 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-03-09T00:17:19.280 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-03-09T00:17:19.282 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-03-09T00:17:19.371 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-03-09T00:17:19.467 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-03-09T00:17:19.479 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-03-09T00:17:19.480 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-03-09T00:17:19.580 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-03-09T00:17:19.665 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-03-09T00:17:19.678 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-03-09T00:17:19.679 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-03-09T00:17:19.795 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-03-09T00:17:19.860 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-03-09T00:17:19.872 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-03-09T00:17:19.883 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-03-09T00:17:20.000 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a 2020-03-09T00:17:20.001 INFO:teuthology.task.ansible.out:23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-03-09T00:17:20.065 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-03-09T00:17:20.077 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-03-09T00:17:20.078 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-03-09T00:17:20.214 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-03-09T00:17:20.278 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-03-09T00:17:20.280 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-03-09T00:17:20.281 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-03-09T00:17:20.429 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-03-09T00:17:20.473 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-03-09T00:17:20.474 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-03-09T00:17:20.476 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-03-09T00:17:20.663 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-03-09T00:17:20.666 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-03-09T00:17:20.667 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-03-09T00:17:20.679 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-03-09T00:17:20.861 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-03-09T00:17:20.863 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-03-09T00:17:20.875 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-03-09T00:17:20.876 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-03-09T00:17:21.058 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-03-09T00:17:21.059 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-03-09T00:17:21.071 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-03-09T00:17:21.094 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-03-09T00:17:21.255 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-03-09T00:17:21.256 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-03-09T00:17:21.289 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-03-09T00:17:21.301 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-03-09T00:17:21.450 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-03-09T00:17:21.451 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-03-09T00:17:21.495 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-03-09T00:17:21.517 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-03-09T00:17:21.654 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-03-09T00:17:21.713 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31e 2020-03-09T00:17:21.714 INFO:teuthology.task.ansible.out:f269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-03-09T00:17:21.722 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-03-09T00:17:21.860 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-03-09T00:17:21.883 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-03-09T00:17:21.895 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-03-09T00:17:21.949 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-03-09T00:17:22.054 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-03-09T00:17:22.087 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-03-09T00:17:22.099 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-03-09T00:17:22.163 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-03-09T00:17:22.258 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-03-09T00:17:22.291 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-03-09T00:17:22.314 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-03-09T00:17:22.377 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-03-09T00:17:22.464 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-03-09T00:17:22.496 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-03-09T00:17:22.508 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-03-09T00:17:22.593 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-03-09T00:17:22.700 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-03-09T00:17:22.701 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-03-09T00:17:22.713 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-03-09T00:17:22.798 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-03-09T00:17:22.904 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-03-09T00:17:22.916 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-03-09T00:17:22.917 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-03-09T00:17:23.014 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-03-09T00:17:23.110 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-03-09T00:17:23.121 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-03-09T00:17:23.123 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-03-09T00:17:23.260 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-03-09T00:17:23.303 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-03-09T00:17:23.315 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-03-09T00:17:23.316 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-03-09T00:17:23.485 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-03-09T00:17:23.507 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-03-09T00:17:23.519 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-03-09T00:17:23.520 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-03-09T00:17:23.711 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-03-09T00:17:23.713 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-03-09T00:17:23.714 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-03-09T00:17:23.715 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-03-09T00:17:23.907 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-03-09T00:17:23.930 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-03-09T00:17:23.931 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-03-09T00:17:23.934 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-03-09T00:17:24.115 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-03-09T00:17:24.139 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-03-09T00:17:24.162 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-03-09T00:17:24.163 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-03-09T00:17:24.323 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-03-09T00:17:24.335 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-03-09T00:17:24.368 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-03-09T00:17:24.395 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-03-09T00:17:24.546 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-03-09T00:17:24.569 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-03-09T00:17:24.628 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-03-09T00:17:24.768 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-03-09T00:17:24.780 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:24.791 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-03-09T00:17:24.842 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-03-09T00:17:24.983 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-03-09T00:17:24.995 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-03-09T00:17:25.008 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-03-09T00:17:25.073 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-03-09T00:17:25.192 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:25.214 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-03-09T00:17:25.226 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:25.284 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:25.392 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-03-09T00:17:25.414 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-03-09T00:17:25.426 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-03-09T00:17:25.495 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-03-09T00:17:25.591 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-03-09T00:17:25.613 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-03-09T00:17:25.625 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-03-09T00:17:25.724 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-03-09T00:17:25.810 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-03-09T00:17:25.811 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-03-09T00:17:25.823 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-03-09T00:17:25.934 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-03-09T00:17:26.009 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-03-09T00:17:26.010 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-03-09T00:17:26.022 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-03-09T00:17:26.139 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-03-09T00:17:26.205 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-03-09T00:17:26.207 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-03-09T00:17:26.218 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-03-09T00:17:26.361 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-03-09T00:17:26.404 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-03-09T00:17:26.405 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:26.418 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-03-09T00:17:26.584 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-03-09T00:17:26.596 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-03-09T00:17:26.598 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-03-09T00:17:26.621 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-03-09T00:17:26.792 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-03-09T00:17:26.804 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:26.805 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-03-09T00:17:26.817 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:27.002 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-03-09T00:17:27.003 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-03-09T00:17:27.023 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:27.198 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-03-09T00:17:27.210 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-03-09T00:17:27.212 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-03-09T00:17:27.234 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-03-09T00:17:27.408 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-03-09T00:17:27.420 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-03-09T00:17:27.421 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-03-09T00:17:27.444 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-03-09T00:17:27.611 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-03-09T00:17:27.624 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-03-09T00:17:27.659 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-03-09T00:17:27.737 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-03-09T00:17:27.801 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-03-09T00:17:27.833 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-03-09T00:17:27.879 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-03-09T00:17:27.944 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-03-09T00:17:28.090 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-03-09T00:17:28.134 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-03-09T00:17:28.147 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-03-09T00:17:28.150 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-03-09T00:17:28.318 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-03-09T00:17:28.329 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-03-09T00:17:28.342 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-03-09T00:17:28.368 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-03-09T00:17:28.540 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-03-09T00:17:28.563 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-03-09T00:17:28.741 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-03-09T00:17:28.742 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-03-09T00:17:28.757 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-03-09T00:17:28.767 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-03-09T00:17:28.937 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-03-09T00:17:28.949 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-03-09T00:17:28.972 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-03-09T00:17:29.098 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-03-09T00:17:29.141 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-03-09T00:17:29.153 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-03-09T00:17:29.176 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-03-09T00:17:29.303 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-03-09T00:17:29.346 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-03-09T00:17:29.358 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-03-09T00:17:29.381 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-03-09T00:17:29.521 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-03-09T00:17:29.596 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-03-09T00:17:29.651 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-03-09T00:17:29.663 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-03-09T00:17:29.749 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-03-09T00:17:29.792 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-03-09T00:17:29.847 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-03-09T00:17:29.870 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-03-09T00:17:29.959 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-03-09T00:17:29.993 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-03-09T00:17:30.059 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:30.071 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-03-09T00:17:30.173 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-03-09T00:17:30.226 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-03-09T00:17:30.291 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-03-09T00:17:30.292 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-03-09T00:17:30.396 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-03-09T00:17:30.446 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:30.521 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:30.523 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-03-09T00:17:30.597 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-03-09T00:17:30.652 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-03-09T00:17:30.718 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-03-09T00:17:30.719 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-03-09T00:17:30.835 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-09T00:17:30.846 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-03-09T00:17:30.921 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-03-09T00:17:30.922 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-03-09T00:17:31.029 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-03-09T00:17:31.051 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-03-09T00:17:31.126 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-03-09T00:17:31.141 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-03-09T00:17:31.238 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-03-09T00:17:31.271 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-03-09T00:17:31.337 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-03-09T00:17:31.351 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-03-09T00:17:31.472 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-03-09T00:17:31.494 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-03-09T00:17:31.534 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-03-09T00:17:31.566 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-03-09T00:17:31.696 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-03-09T00:17:31.697 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-03-09T00:17:31.741 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-03-09T00:17:31.778 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-03-09T00:17:31.920 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-03-09T00:17:31.932 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-03-09T00:17:31.944 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-03-09T00:17:31.989 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-03-09T00:17:32.139 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-03-09T00:17:32.140 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-03-09T00:17:32.152 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-03-09T00:17:32.206 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-03-09T00:17:32.343 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-03-09T00:17:32.366 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-03-09T00:17:32.378 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-03-09T00:17:32.411 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-03-09T00:17:32.549 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-03-09T00:17:32.575 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-03-09T00:17:32.591 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-03-09T00:17:32.614 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-03-09T00:17:32.742 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-03-09T00:17:32.807 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-03-09T00:17:32.819 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-03-09T00:17:32.821 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-03-09T00:17:32.949 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-03-09T00:17:33.015 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-03-09T00:17:33.036 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-03-09T00:17:33.060 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-03-09T00:17:33.148 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-03-09T00:17:33.225 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-03-09T00:17:33.238 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-03-09T00:17:33.273 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-03-09T00:17:33.348 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-03-09T00:17:33.423 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-03-09T00:17:33.435 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-03-09T00:17:33.479 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-03-09T00:17:33.544 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-03-09T00:17:33.622 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-03-09T00:17:33.635 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-03-09T00:17:33.710 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-03-09T00:17:33.753 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-03-09T00:17:33.829 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-03-09T00:17:33.841 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-03-09T00:17:33.930 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-03-09T00:17:33.952 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-03-09T00:17:34.018 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-03-09T00:17:34.041 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-03-09T00:17:34.149 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-03-09T00:17:34.150 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-03-09T00:17:34.216 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-03-09T00:17:34.239 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-03-09T00:17:34.358 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-03-09T00:17:34.360 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-03-09T00:17:34.414 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-03-09T00:17:34.447 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-03-09T00:17:34.555 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-03-09T00:17:34.567 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-03-09T00:17:34.620 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-03-09T00:17:34.654 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-03-09T00:17:34.759 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-03-09T00:17:34.782 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-03-09T00:17:34.815 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-03-09T00:17:34.858 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-03-09T00:17:34.964 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-03-09T00:17:34.997 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-03-09T00:17:35.020 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-03-09T00:17:35.053 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-03-09T00:17:35.159 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-03-09T00:17:35.202 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-03-09T00:17:35.224 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-03-09T00:17:35.246 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-03-09T00:17:35.354 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-03-09T00:17:35.418 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-03-09T00:17:35.419 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-03-09T00:17:35.453 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-03-09T00:17:35.561 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-03-09T00:17:35.615 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-03-09T00:17:35.628 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-03-09T00:17:35.650 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-03-09T00:17:35.758 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-03-09T00:17:35.823 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-03-09T00:17:35.845 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-03-09T00:17:35.857 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-03-09T00:17:35.973 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-03-09T00:17:36.028 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-03-09T00:17:36.067 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-03-09T00:17:36.068 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-03-09T00:17:36.176 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-03-09T00:17:36.230 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-03-09T00:17:36.263 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-03-09T00:17:36.286 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-03-09T00:17:36.404 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-03-09T00:17:36.448 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-03-09T00:17:36.482 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-03-09T00:17:36.494 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-03-09T00:17:36.606 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-03-09T00:17:36.649 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-03-09T00:17:36.683 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-03-09T00:17:36.721 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-03-09T00:17:36.829 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-03-09T00:17:36.852 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-03-09T00:17:36.876 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-03-09T00:17:36.944 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-03-09T00:17:37.030 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-03-09T00:17:37.053 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-03-09T00:17:37.087 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-03-09T00:17:37.162 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-03-09T00:17:37.238 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-03-09T00:17:37.283 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-03-09T00:17:37.285 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-03-09T00:17:37.395 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-03-09T00:17:37.449 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-03-09T00:17:37.482 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-03-09T00:17:37.506 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-03-09T00:17:37.603 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-03-09T00:17:37.650 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-03-09T00:17:37.694 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-03-09T00:17:37.706 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-03-09T00:17:37.844 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-03-09T00:17:37.867 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:37.912 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-03-09T00:17:37.913 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-03-09T00:17:38.052 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-03-09T00:17:38.065 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-03-09T00:17:38.110 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-03-09T00:17:38.122 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-03-09T00:17:38.263 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-03-09T00:17:38.265 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-03-09T00:17:38.333 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:38.334 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:38.474 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-03-09T00:17:38.488 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-03-09T00:17:38.531 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-03-09T00:17:38.543 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-03-09T00:17:38.672 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-03-09T00:17:38.705 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-03-09T00:17:38.740 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-03-09T00:17:38.753 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-03-09T00:17:38.865 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-03-09T00:17:38.919 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-03-09T00:17:38.941 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-03-09T00:17:38.984 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-03-09T00:17:39.070 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-03-09T00:17:39.134 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-03-09T00:17:39.146 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-09T00:17:39.190 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-03-09T00:17:39.264 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-03-09T00:17:39.339 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-03-09T00:17:39.362 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-03-09T00:17:39.390 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-03-09T00:17:39.465 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-03-09T00:17:39.540 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-03-09T00:17:39.572 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-03-09T00:17:39.584 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-03-09T00:17:39.736 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-03-09T00:17:39.790 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-03-09T00:17:39.791 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-03-09T00:17:39.948 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-03-09T00:17:39.995 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-03-09T00:17:39.997 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-03-09T00:17:40.229 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-03-09T00:17:40.430 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-03-09T00:17:40.653 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-03-09T00:17:40.874 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-03-09T00:17:40.907 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "name": "khiremat", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-03-09T00:17:41.045 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:41 +0000 (0:00:23.815) 0:00:33.018 ********** 2020-03-09T00:17:41.072 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-09T00:17:41.109 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:41 +0000 (0:00:00.064) 0:00:33.082 ********** 2020-03-09T00:17:41.305 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-09T00:17:41.309 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-03-09T00:17:41.311 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-03-09T00:17:41.312 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-03-09T00:17:41.313 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-03-09T00:17:41.315 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-03-09T00:17:41.316 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-03-09T00:17:41.318 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d 2020-03-09T00:17:41.318 INFO:teuthology.task.ansible.out:49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-03-09T00:17:41.319 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-03-09T00:17:41.321 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-03-09T00:17:41.322 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-03-09T00:17:41.323 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}]}, "changed": false} 2020-03-09T00:17:41.363 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:41 +0000 (0:00:00.253) 0:00:33.336 ********** 2020-03-09T00:17:41.454 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-09T00:17:41.454 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "name": "khiremat", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-03-09T00:17:41.685 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "name": "khiremat", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-03-09T00:17:41.796 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:41 +0000 (0:00:00.432) 0:00:33.768 ********** 2020-03-09T00:17:41.829 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:41 +0000 (0:00:00.033) 0:00:33.802 ********** 2020-03-09T00:17:41.835 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-09T00:17:41.888 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:41 +0000 (0:00:00.058) 0:00:33.860 ********** 2020-03-09T00:17:41.939 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:41 +0000 (0:00:00.051) 0:00:33.912 ********** 2020-03-09T00:17:41.948 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-09T00:17:41.952 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-03-09T00:17:41.954 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-03-09T00:17:41.955 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-03-09T00:17:41.957 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-03-09T00:17:41.958 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-03-09T00:17:41.959 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-03-09T00:17:41.961 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d 2020-03-09T00:17:41.961 INFO:teuthology.task.ansible.out:49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-03-09T00:17:41.962 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-03-09T00:17:41.964 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-03-09T00:17:41.965 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-03-09T00:17:41.967 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}]}, "changed": false} 2020-03-09T00:17:42.007 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.067) 0:00:33.979 ********** 2020-03-09T00:17:42.020 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-03-09T00:17:42.021 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-03-09T00:17:42.023 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-03-09T00:17:42.024 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-03-09T00:17:42.026 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-03-09T00:17:42.027 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-03-09T00:17:42.029 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d 2020-03-09T00:17:42.029 INFO:teuthology.task.ansible.out:49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-03-09T00:17:42.030 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-03-09T00:17:42.032 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-03-09T00:17:42.033 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-03-09T00:17:42.035 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}]}, "changed": false} 2020-03-09T00:17:42.073 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.066) 0:00:34.046 ********** 2020-03-09T00:17:42.511 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-09T00:17:42.513 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com -> localhost] => {"after": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "attempts": 1, "before": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "changed": false, "remote_url_changed": false} 2020-03-09T00:17:42.549 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.476) 0:00:34.522 ********** 2020-03-09T00:17:42.577 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-09T00:17:42.577 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "name": "khiremat", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-03-09T00:17:42.677 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-09T00:17:42.678 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com -> localhost] => {"after": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "attempts": 1, "before": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "changed": false, "remote_url_changed": false} 2020-03-09T00:17:42.716 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.166) 0:00:34.688 ********** 2020-03-09T00:17:42.749 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.033) 0:00:34.722 ********** 2020-03-09T00:17:42.757 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com -> localhost] => {"after": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "attempts": 1, "before": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "changed": false, "remote_url_changed": false} 2020-03-09T00:17:42.759 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-09T00:17:42.798 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.049) 0:00:34.771 ********** 2020-03-09T00:17:42.830 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.031) 0:00:34.803 ********** 2020-03-09T00:17:42.853 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-09T00:17:42.857 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-03-09T00:17:42.858 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-03-09T00:17:42.860 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-03-09T00:17:42.861 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-03-09T00:17:42.863 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-03-09T00:17:42.864 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-03-09T00:17:42.865 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-03-09T00:17:42.867 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, 2020-03-09T00:17:42.867 INFO:teuthology.task.ansible.out:{"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-03-09T00:17:42.868 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-03-09T00:17:42.870 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-03-09T00:17:42.871 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}]}, "changed": false} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, 2020-03-09T00:17:42.873 INFO:teuthology.task.ansible.out: "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-03-09T00:17:42.913 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:42 +0000 (0:00:00.083) 0:00:34.886 ********** 2020-03-09T00:17:43.048 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-03-09T00:17:43.091 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-03-09T00:17:43.134 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-03-09T00:17:43.303 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-03-09T00:17:43.315 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-03-09T00:17:43.360 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-03-09T00:17:43.547 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-03-09T00:17:43.548 INFO:teuthology.task.ansible.out:lidate_certs": true} 2020-03-09T00:17:43.559 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-03-09T00:17:43.581 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-03-09T00:17:43.616 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-09T00:17:43.625 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com -> localhost] => {"after": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "attempts": 1, "before": "d9f2fa52162dd365bfcfaae13ab61fb6caff6e16", "changed": false, "remote_url_changed": false} 2020-03-09T00:17:43.652 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:17:43 +0000 (0:00:00.738) 0:00:35.625 ********** 2020-03-09T00:17:43.784 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:17:43.786 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-03-09T00:17:43.787 INFO:teuthology.task.ansible.out:lidate_certs": true} ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_cer 2020-03-09T00:17:43.790 INFO:teuthology.task.ansible.out:ts": true} 2020-03-09T00:17:43.799 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-03-09T00:17:43.800 INFO:teuthology.task.ansible.out:lidate_certs": true} 2020-03-09T00:17:43.982 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-03-09T00:17:44.026 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-03-09T00:17:44.048 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-03-09T00:17:44.049 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-03-09T00:17:44.050 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-03-09T00:17:44.237 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-03-09T00:17:44.260 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-03-09T00:17:44.262 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-03-09T00:17:44.271 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-03-09T00:17:44.272 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-03-09T00:17:44.274 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-03-09T00:17:44.462 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-03-09T00:17:44.485 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-03-09T00:17:44.487 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-03-09T00:17:44.488 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-03-09T00:17:44.499 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.699 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-03-09T00:17:44.700 INFO:teuthology.task.ansible.out:lidate_certs": true} ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-03-09T00:17:44.702 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-03-09T00:17:44.712 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.724 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.725 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-03-09T00:17:44.726 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.737 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.748 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.938 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-03-09T00:17:44.940 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC 2020-03-09T00:17:44.941 INFO:teuthology.task.ansible.out:1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3 2020-03-09T00:17:44.943 INFO:teuthology.task.ansible.out:NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-03-09T00:17:44.952 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.953 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.954 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.965 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.966 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.968 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:44.969 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-03-09T00:17:45.182 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-03-09T00:17:45.184 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-03-09T00:17:45.185 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-03-09T00:17:45.197 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-03-09T00:17:45.198 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-03-09T00:17:45.408 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-03-09T00:17:45.410 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7M 2020-03-09T00:17:45.416 INFO:teuthology.task.ansible.out:wYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwA 2020-03-09T00:17:45.418 INFO:teuthology.task.ansible.out:AAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-03-09T00:17:45.421 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-03-09T00:17:45.633 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RD 2020-03-09T00:17:45.635 INFO:teuthology.task.ansible.out:rc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-03-09T00:17:45.636 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2 2020-03-09T00:17:45.638 INFO:teuthology.task.ansible.out:X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-03-09T00:17:45.638 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2 2020-03-09T00:17:45.639 INFO:teuthology.task.ansible.out:X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-03-09T00:17:45.671 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-03-09T00:17:45.682 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:45.853 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-03-09T00:17:45.854 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-03-09T00:17:45.876 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-03-09T00:17:45.920 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f 2020-03-09T00:17:45.921 INFO:teuthology.task.ansible.out:80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-03-09T00:17:45.931 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toI 2020-03-09T00:17:45.932 INFO:teuthology.task.ansible.out:j9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:45.943 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:45.960 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:17:46.077 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-03-09T00:17:46.099 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-03-09T00:17:46.100 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-03-09T00:17:46.197 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-03-09T00:17:46.292 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-03-09T00:17:46.314 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-03-09T00:17:46.316 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-03-09T00:17:46.440 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-03-09T00:17:46.442 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-03-09T00:17:46.515 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-03-09T00:17:46.526 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-03-09T00:17:46.538 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-03-09T00:17:46.665 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-03-09T00:17:46.729 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-03-09T00:17:46.751 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-03-09T00:17:46.752 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-03-09T00:17:46.754 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-03-09T00:17:46.889 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-03-09T00:17:46.954 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-03-09T00:17:46.956 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-03-09T00:17:46.976 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-03-09T00:17:46.978 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa A 2020-03-09T00:17:46.979 INFO:teuthology.task.ansible.out:AAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-03-09T00:17:47.136 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-03-09T00:17:47.181 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-03-09T00:17:47.183 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-03-09T00:17:47.204 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-03-09T00:17:47.205 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-03-09T00:17:47.206 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-03-09T00:17:47.366 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-03-09T00:17:47.409 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-03-09T00:17:47.420 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-03-09T00:17:47.431 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-03-09T00:17:47.602 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-03-09T00:17:47.624 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-03-09T00:17:47.646 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-03-09T00:17:47.658 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-03-09T00:17:47.659 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-03-09T00:17:47.844 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-03-09T00:17:47.855 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-03-09T00:17:47.857 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-03-09T00:17:47.878 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-03-09T00:17:47.879 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-03-09T00:17:48.080 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-03-09T00:17:48.081 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-03-09T00:17:48.103 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-03-09T00:17:48.104 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-03-09T00:17:48.320 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-03-09T00:17:48.322 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": 2020-03-09T00:17:48.334 INFO:teuthology.task.ansible.out:true} 2020-03-09T00:17:48.335 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-03-09T00:17:48.336 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-03-09T00:17:48.559 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-03-09T00:17:48.560 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-03-09T00:17:48.561 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-03-09T00:17:48.572 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-03-09T00:17:48.782 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-03-09T00:17:48.798 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-03-09T00:17:48.806 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-03-09T00:17:48.807 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-03-09T00:17:49.029 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-03-09T00:17:49.033 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-03-09T00:17:49.035 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-03-09T00:17:49.037 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1 2020-03-09T00:17:49.038 INFO:teuthology.task.ansible.out:VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-03-09T00:17:49.067 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-03-09T00:17:49.070 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-03-09T00:17:49.262 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-03-09T00:17:49.263 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRv 2020-03-09T00:17:49.265 INFO:teuthology.task.ansible.out:St nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8 2020-03-09T00:17:49.266 INFO:teuthology.task.ansible.out:b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-03-09T00:17:49.306 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-03-09T00:17:49.481 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-03-09T00:17:49.493 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-03-09T00:17:49.495 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-03-09T00:17:49.548 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-03-09T00:17:49.706 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-03-09T00:17:49.718 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-03-09T00:17:49.719 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-03-09T00:17:49.783 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-03-09T00:17:49.921 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-03-09T00:17:49.932 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-03-09T00:17:49.944 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-03-09T00:17:50.018 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-03-09T00:17:50.149 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-03-09T00:17:50.161 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-03-09T00:17:50.162 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-03-09T00:17:50.247 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-03-09T00:17:50.375 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-03-09T00:17:50.377 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": 2020-03-09T00:17:50.379 INFO:teuthology.task.ansible.out: "present", "user": "jdillaman", "validate_certs": true} 2020-03-09T00:17:50.494 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-03-09T00:17:50.495 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-03-09T00:17:50.600 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-03-09T00:17:50.601 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:50.604 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-03-09T00:17:50.754 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-03-09T00:17:50.819 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-03-09T00:17:50.821 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:50.822 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdP 2020-03-09T00:17:50.824 INFO:teuthology.task.ansible.out:Buld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:50.992 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-03-09T00:17:51.035 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-03-09T00:17:51.047 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-03-09T00:17:51.062 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-03-09T00:17:51.232 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-03-09T00:17:51.254 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-03-09T00:17:51.265 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-03-09T00:17:51.279 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-03-09T00:17:51.281 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-03-09T00:17:51.468 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-03-09T00:17:51.479 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-03-09T00:17:51.481 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-03-09T00:17:51.502 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-03-09T00:17:51.503 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZ 2020-03-09T00:17:51.505 INFO:teuthology.task.ansible.out:jNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-03-09T00:17:51.693 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-03-09T00:17:51.704 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-03-09T00:17:51.705 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-03-09T00:17:51.727 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-03-09T00:17:51.728 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-03-09T00:17:51.738 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-03-09T00:17:51.928 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-03-09T00:17:51.929 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-03-09T00:17:51.951 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-03-09T00:17:51.963 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-03-09T00:17:52.152 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-03-09T00:17:52.163 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:52.185 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-03-09T00:17:52.197 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-03-09T00:17:52.408 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-03-09T00:17:52.409 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-03-09T00:17:52.411 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-03-09T00:17:52.422 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-03-09T00:17:52.424 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-09T00:17:52.632 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-03-09T00:17:52.634 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40Gul 2020-03-09T00:17:52.636 INFO:teuthology.task.ansible.out:uQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-09T00:17:52.645 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-03-09T00:17:52.646 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-03-09T00:17:52.647 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-03-09T00:17:52.649 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-03-09T00:17:52.866 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-03-09T00:17:52.868 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-03-09T00:17:52.870 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861a 2020-03-09T00:17:52.871 INFO:teuthology.task.ansible.out:d1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p 2020-03-09T00:17:52.872 INFO:teuthology.task.ansible.out:/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-03-09T00:17:52.892 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-03-09T00:17:52.894 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-03-09T00:17:53.084 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-03-09T00:17:53.127 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-03-09T00:17:53.149 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-03-09T00:17:53.151 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-03-09T00:17:53.276 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-03-09T00:17:53.309 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-03-09T00:17:53.363 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-03-09T00:17:53.396 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-03-09T00:17:53.513 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-03-09T00:17:53.525 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-03-09T00:17:53.589 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-03-09T00:17:53.632 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-03-09T00:17:53.737 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-03-09T00:17:53.748 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-03-09T00:17:53.823 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-03-09T00:17:53.876 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-03-09T00:17:53.964 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-03-09T00:17:53.988 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-03-09T00:17:54.044 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-03-09T00:17:54.110 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-03-09T00:17:54.112 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-09T00:17:54.189 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-03-09T00:17:54.214 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-03-09T00:17:54.268 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-03-09T00:17:54.353 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-03-09T00:17:54.354 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-03-09T00:17:54.357 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-03-09T00:17:54.428 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-03-09T00:17:54.443 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-03-09T00:17:54.501 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-03-09T00:17:54.597 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-03-09T00:17:54.663 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-03-09T00:17:54.675 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-03-09T00:17:54.729 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-03-09T00:17:54.848 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-03-09T00:17:54.883 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-03-09T00:17:54.884 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-03-09T00:17:54.895 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-03-09T00:17:54.897 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-03-09T00:17:54.960 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-03-09T00:17:54.962 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-03-09T00:17:55.078 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-03-09T00:17:55.111 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-03-09T00:17:55.134 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-03-09T00:17:55.188 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-03-09T00:17:55.318 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-03-09T00:17:55.340 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-03-09T00:17:55.351 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-03-09T00:17:55.420 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-03-09T00:17:55.572 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-03-09T00:17:55.583 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-03-09T00:17:55.585 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJp 2020-03-09T00:17:55.586 INFO:teuthology.task.ansible.out:ozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9c 2020-03-09T00:17:55.588 INFO:teuthology.task.ansible.out:zHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-03-09T00:17:55.638 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-03-09T00:17:55.639 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-03-09T00:17:55.795 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-03-09T00:17:55.807 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-03-09T00:17:55.809 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-03-09T00:17:55.852 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-03-09T00:17:56.021 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-03-09T00:17:56.032 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-03-09T00:17:56.055 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-03-09T00:17:56.088 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-03-09T00:17:56.248 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-03-09T00:17:56.251 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-03-09T00:17:56.289 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-03-09T00:17:56.306 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-03-09T00:17:56.464 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:56.466 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:56.519 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-03-09T00:17:56.521 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-03-09T00:17:56.531 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:56.689 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-03-09T00:17:56.690 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-03-09T00:17:56.749 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-03-09T00:17:56.752 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-03-09T00:17:56.921 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-03-09T00:17:56.922 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-03-09T00:17:56.986 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-03-09T00:17:56.997 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-03-09T00:17:57.144 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-03-09T00:17:57.166 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-03-09T00:17:57.231 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-03-09T00:17:57.232 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-03-09T00:17:57.243 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-03-09T00:17:57.373 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-03-09T00:17:57.385 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-03-09T00:17:57.448 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-03-09T00:17:57.461 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-03-09T00:17:57.591 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-03-09T00:17:57.604 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-03-09T00:17:57.679 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-03-09T00:17:57.692 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-03-09T00:17:57.820 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-03-09T00:17:57.841 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-03-09T00:17:57.906 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-03-09T00:17:57.927 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-03-09T00:17:58.037 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-03-09T00:17:58.067 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-03-09T00:17:58.121 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-03-09T00:17:58.164 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-09T00:17:58.270 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-03-09T00:17:58.303 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-03-09T00:17:58.346 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-03-09T00:17:58.390 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-03-09T00:17:58.496 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-03-09T00:17:58.540 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-03-09T00:17:58.562 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-03-09T00:17:58.626 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-03-09T00:17:58.721 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-03-09T00:17:58.776 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-03-09T00:17:58.788 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-03-09T00:17:58.862 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-03-09T00:17:58.937 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-03-09T00:17:58.990 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-03-09T00:17:59.012 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-03-09T00:17:59.107 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-03-09T00:17:59.160 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-03-09T00:17:59.214 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-03-09T00:17:59.227 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-03-09T00:17:59.359 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-03-09T00:17:59.388 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-03-09T00:17:59.433 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-03-09T00:17:59.455 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-03-09T00:17:59.584 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-03-09T00:17:59.617 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-03-09T00:17:59.649 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-03-09T00:17:59.690 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-03-09T00:17:59.818 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-03-09T00:17:59.840 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-03-09T00:17:59.896 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-03-09T00:17:59.919 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-03-09T00:18:00.055 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-03-09T00:18:00.056 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-03-09T00:18:00.113 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-03-09T00:18:00.146 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-03-09T00:18:00.272 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-03-09T00:18:00.284 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-03-09T00:18:00.348 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-03-09T00:18:00.370 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-03-09T00:18:00.518 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-03-09T00:18:00.519 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-03-09T00:18:00.562 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-03-09T00:18:00.597 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-03-09T00:18:00.736 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-03-09T00:18:00.748 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-03-09T00:18:00.792 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-03-09T00:18:00.824 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-03-09T00:18:00.971 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-03-09T00:18:00.976 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-03-09T00:18:01.009 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-03-09T00:18:01.042 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-03-09T00:18:01.191 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-03-09T00:18:01.213 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-03-09T00:18:01.236 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-03-09T00:18:01.279 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-03-09T00:18:01.437 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-03-09T00:18:01.438 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-03-09T00:18:01.448 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-03-09T00:18:01.451 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-03-09T00:18:01.452 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-03-09T00:18:01.515 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-03-09T00:18:01.516 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-03-09T00:18:01.663 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-03-09T00:18:01.665 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-03-09T00:18:01.697 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-03-09T00:18:01.699 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyAD 2020-03-09T00:18:01.700 INFO:teuthology.task.ansible.out:xrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-03-09T00:18:01.743 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-03-09T00:18:01.744 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-03-09T00:18:01.885 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-03-09T00:18:01.918 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-03-09T00:18:01.933 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-03-09T00:18:01.974 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-03-09T00:18:02.125 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-03-09T00:18:02.148 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-03-09T00:18:02.160 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-03-09T00:18:02.203 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-03-09T00:18:02.330 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-03-09T00:18:02.362 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-03-09T00:18:02.395 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-03-09T00:18:02.417 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-03-09T00:18:02.554 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-03-09T00:18:02.586 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-03-09T00:18:02.630 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-03-09T00:18:02.641 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-03-09T00:18:02.789 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-03-09T00:18:02.811 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-03-09T00:18:02.865 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-03-09T00:18:02.876 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-03-09T00:18:03.014 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-03-09T00:18:03.025 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-03-09T00:18:03.090 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-03-09T00:18:03.102 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-03-09T00:18:03.250 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-03-09T00:18:03.251 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-03-09T00:18:03.315 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-03-09T00:18:03.337 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-03-09T00:18:03.339 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-03-09T00:18:03.467 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-03-09T00:18:03.478 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-03-09T00:18:03.532 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-03-09T00:18:03.576 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-03-09T00:18:03.577 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-03-09T00:18:03.682 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-03-09T00:18:03.714 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-03-09T00:18:03.768 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-03-09T00:18:03.822 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "ke 2020-03-09T00:18:03.823 INFO:teuthology.task.ansible.out:y_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-03-09T00:18:03.907 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-03-09T00:18:03.951 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-03-09T00:18:03.995 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-03-09T00:18:04.051 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-03-09T00:18:04.126 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-03-09T00:18:04.159 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-03-09T00:18:04.223 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-03-09T00:18:04.288 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-03-09T00:18:04.353 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-03-09T00:18:04.375 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-03-09T00:18:04.445 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-03-09T00:18:04.531 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-03-09T00:18:04.585 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-03-09T00:18:04.621 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-03-09T00:18:04.665 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-03-09T00:18:04.760 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-03-09T00:18:04.815 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-03-09T00:18:04.827 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-03-09T00:18:04.888 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-03-09T00:18:04.996 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-03-09T00:18:05.040 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-03-09T00:18:05.042 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-09T00:18:05.052 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-03-09T00:18:05.054 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-09T00:18:05.096 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-03-09T00:18:05.097 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-09T00:18:05.234 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-03-09T00:18:05.258 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-03-09T00:18:05.281 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-03-09T00:18:05.314 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-03-09T00:18:05.483 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-03-09T00:18:05.494 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-03-09T00:18:05.506 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-03-09T00:18:05.541 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-03-09T00:18:05.710 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-03-09T00:18:05.721 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-03-09T00:18:05.722 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-03-09T00:18:05.755 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-03-09T00:18:05.952 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-03-09T00:18:05.953 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-03-09T00:18:05.955 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-03-09T00:18:05.977 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-03-09T00:18:06.175 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-03-09T00:18:06.186 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-03-09T00:18:06.198 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-03-09T00:18:06.199 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-03-09T00:18:06.400 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-03-09T00:18:06.422 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-03-09T00:18:06.433 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-03-09T00:18:06.465 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-03-09T00:18:06.627 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-03-09T00:18:06.660 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-03-09T00:18:06.661 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-03-09T00:18:06.683 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-03-09T00:18:06.863 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-03-09T00:18:06.896 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-03-09T00:18:06.897 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-03-09T00:18:06.899 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-03-09T00:18:07.099 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-03-09T00:18:07.199 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-03-09T00:18:07.246 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-03-09T00:18:07.247 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-03-09T00:18:07.249 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-09T00:18:07.327 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:07 +0000 (0:00:23.674) 0:00:59.299 ********** 2020-03-09T00:18:07.362 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:07 +0000 (0:00:00.034) 0:00:59.334 ********** 2020-03-09T00:18:07.406 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:07 +0000 (0:00:00.044) 0:00:59.379 ********** 2020-03-09T00:18:07.412 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:07.414 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.415 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.417 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:07.417 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-03-09T00:18:07.419 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.420 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.422 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.441 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.442 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.442 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.444 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.454 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.455 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.459 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.468 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.470 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.480 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.482 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.493 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.504 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.516 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.516 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.625 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:07.638 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-03-09T00:18:07.671 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:07.672 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-03-09T00:18:07.683 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.705 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-03-09T00:18:07.727 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.760 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-03-09T00:18:07.772 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:07.856 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:07.858 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-03-09T00:18:07.899 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:07.902 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-03-09T00:18:07.944 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-03-09T00:18:07.986 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-03-09T00:18:08.092 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:08.094 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-03-09T00:18:08.114 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:08.115 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-03-09T00:18:08.169 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-03-09T00:18:08.201 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-03-09T00:18:08.318 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:08.320 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-03-09T00:18:08.330 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:08.330 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-03-09T00:18:08.341 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.352 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.363 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.374 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.375 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.386 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-03-09T00:18:08.387 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.398 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.409 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.410 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.425 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7 2020-03-09T00:18:08.426 INFO:teuthology.task.ansible.out:\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGl 2020-03-09T00:18:08.428 INFO:teuthology.task.ansible.out:oB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.437 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.437 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.449 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-03-09T00:18:08.450 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.451 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.451 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.462 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.463 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.466 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.475 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.476 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.477 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.478 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.480 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.490 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.492 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.502 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.503 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.505 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.515 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.516 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.516 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.518 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.529 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.530 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.532 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.542 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.542 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.544 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.544 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.555 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.557 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.558 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:08.558 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9w 2020-03-09T00:18:08.560 INFO:teuthology.task.ansible.out:TnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTf 2020-03-09T00:18:08.561 INFO:teuthology.task.ansible.out:DCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.571 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.573 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.575 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.584 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.587 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.587 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.600 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.602 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.612 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.612 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.614 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.614 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.625 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.626 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.630 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.640 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.642 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.653 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.654 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.654 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.656 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.666 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.668 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.669 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.680 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.681 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.683 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.694 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.696 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.698 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.708 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.709 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.710 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.711 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.722 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.723 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.724 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.736 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.738 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.739 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.749 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.865 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var 2020-03-09T00:18:08.866 INFO:teuthology.task.ansible.out:": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name" 2020-03-09T00:18:08.868 INFO:teuthology.task.ansible.out:: "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC 2020-03-09T00:18:08.869 INFO:teuthology.task.ansible.out:1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={ 2020-03-09T00:18:08.871 INFO:teuthology.task.ansible.out:u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vsha 2020-03-09T00:18:08.872 INFO:teuthology.task.ansible.out:nkar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076a 2020-03-09T00:18:08.874 INFO:teuthology.task.ansible.out:e7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', 2020-03-09T00:18:08.876 INFO:teuthology.task.ansible.out: u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: 2020-03-09T00:18:08.877 INFO:teuthology.task.ansible.out: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@cent 2020-03-09T00:18:08.879 INFO:teuthology.task.ansible.out:os TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.890 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.891 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.892 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.893 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.894 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.904 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.905 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.906 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.917 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.918 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.919 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.930 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.930 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.932 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.942 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.943 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.945 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.945 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.956 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.957 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.958 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.969 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.970 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.971 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.983 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.984 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.985 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.996 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.997 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:08.999 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.000 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.011 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.012 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.013 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.013 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.025 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.026 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.026 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.027 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.039 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.040 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.042 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.043 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.045 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:09.046 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-03-09T00:18:09.046 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:09.048 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.049 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.060 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.061 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.063 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.074 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.074 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.076 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.087 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.089 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.090 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.092 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.092 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.093 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.106 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.109 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.118 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.119 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.120 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.131 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.132 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.134 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.134 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.136 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.147 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.148 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.149 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.151 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.152 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.163 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.164 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.170 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.177 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.178 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.189 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.190 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.191 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.203 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.203 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.205 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.216 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.218 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.219 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.232 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.232 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.258 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.259 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.260 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.261 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.262 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.264 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.322 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:09 +0000 (0:00:01.915) 0:01:01.294 ********** 2020-03-09T00:18:09.331 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.333 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.334 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-09T00:18:09.336 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-03-09T00:18:09.405 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:09.417 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.417 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.419 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.419 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.421 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.421 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.422 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.423 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.478 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:09 +0000 (0:00:00.155) 0:01:01.450 ********** 2020-03-09T00:18:09.513 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:09 +0000 (0:00:00.034) 0:01:01.485 ********** 2020-03-09T00:18:09.551 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:09 +0000 (0:00:00.038) 0:01:01.523 ********** 2020-03-09T00:18:09.594 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:09 +0000 (0:00:00.043) 0:01:01.566 ********** 2020-03-09T00:18:09.599 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.601 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.602 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.603 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d2 2020-03-09T00:18:09.605 INFO:teuthology.task.ansible.out:49450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.606 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.647 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:09 +0000 (0:00:00.052) 0:01:01.619 ********** 2020-03-09T00:18:09.652 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-09T00:18:09.653 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.691 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:09 +0000 (0:00:00.044) 0:01:01.664 ********** 2020-03-09T00:18:09.750 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:09.753 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-03-09T00:18:09.847 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:09.848 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-03-09T00:18:09.860 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:09.882 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:09.883 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-03-09T00:18:09.926 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-03-09T00:18:09.938 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-03-09T00:18:10.077 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-03-09T00:18:10.089 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:10.090 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-03-09T00:18:10.123 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:10.125 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-03-09T00:18:10.135 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-03-09T00:18:10.273 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-03-09T00:18:10.316 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:10.317 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-03-09T00:18:10.319 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi148.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-03-09T00:18:10.332 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-03-09T00:18:10.474 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-03-09T00:18:10.522 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-03-09T00:18:10.524 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-03-09T00:18:10.559 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:10.559 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-03-09T00:18:10.571 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.583 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.584 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.595 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.606 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.607 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.618 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.631 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.633 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.657 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.658 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.663 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.664 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:10.664 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-03-09T00:18:10.666 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.677 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.679 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.690 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.703 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.704 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.716 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.717 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:10.717 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-03-09T00:18:10.719 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-03-09T00:18:10.720 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.732 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.743 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.744 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.755 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.766 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.767 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.777 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.789 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.789 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.800 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.811 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.822 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.823 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.834 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.845 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.846 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.857 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.868 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.869 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi139.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-03-09T00:18:10.870 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.891 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.902 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.904 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.916 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.916 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:10.917 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-03-09T00:18:10.937 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:10.938 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi104.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-03-09T00:18:10.940 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.950 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.951 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.962 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.973 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.983 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:10.994 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.006 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.006 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.017 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.028 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.039 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.050 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.051 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.062 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:11.063 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.074 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.075 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.086 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.097 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.110 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.125 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:11.127 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-03-09T00:18:11.128 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-09T00:18:11.133 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.133 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:11.134 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-03-09T00:18:11.135 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.146 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.147 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.158 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.170 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.170 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.182 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.193 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.204 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.215 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.216 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.227 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.238 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.249 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.250 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.261 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.272 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi139.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-03-09T00:18:11.274 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.284 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.284 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.301 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.308 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.320 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.321 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi148.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-03-09T00:18:11.323 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-03-09T00:18:11.324 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.334 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.345 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.346 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.357 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.368 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.379 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.380 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.390 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.460 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:11 +0000 (0:00:01.768) 0:01:03.432 ********** 2020-03-09T00:18:11.465 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:11.466 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-03-09T00:18:11.489 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-09T00:18:11.491 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:11.548 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:11 +0000 (0:00:00.088) 0:01:03.521 ********** 2020-03-09T00:18:11.553 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:11.554 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-03-09T00:18:11.555 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-03-09T00:18:11.647 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-03-09T00:18:11.721 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-03-09T00:18:11.722 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-03-09T00:18:11.806 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-03-09T00:18:11.839 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-03-09T00:18:11.924 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-03-09T00:18:11.936 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-03-09T00:18:12.011 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-03-09T00:18:12.032 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-03-09T00:18:12.128 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-03-09T00:18:12.139 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-03-09T00:18:12.213 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-03-09T00:18:12.246 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-03-09T00:18:12.331 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-03-09T00:18:12.332 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-03-09T00:18:12.416 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-03-09T00:18:12.438 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-03-09T00:18:12.512 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-03-09T00:18:12.524 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-03-09T00:18:12.631 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-03-09T00:18:12.633 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-03-09T00:18:12.717 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-03-09T00:18:12.718 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-03-09T00:18:12.835 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-03-09T00:18:12.837 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-03-09T00:18:12.909 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-03-09T00:18:12.910 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-03-09T00:18:13.018 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-03-09T00:18:13.040 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-03-09T00:18:13.093 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-03-09T00:18:13.105 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-03-09T00:18:13.225 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-03-09T00:18:13.258 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-03-09T00:18:13.300 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-03-09T00:18:13.302 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-03-09T00:18:13.407 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-03-09T00:18:13.471 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-03-09T00:18:13.493 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-03-09T00:18:13.504 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-03-09T00:18:13.610 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-03-09T00:18:13.676 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-03-09T00:18:13.688 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-03-09T00:18:13.690 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-03-09T00:18:13.805 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-03-09T00:18:13.882 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-03-09T00:18:13.886 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-03-09T00:18:13.888 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-03-09T00:18:13.994 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-03-09T00:18:14.089 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-03-09T00:18:14.112 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-03-09T00:18:14.123 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-03-09T00:18:14.198 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-03-09T00:18:14.292 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-03-09T00:18:14.304 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-03-09T00:18:14.329 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-03-09T00:18:14.404 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-03-09T00:18:14.485 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-03-09T00:18:14.507 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-03-09T00:18:14.539 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-03-09T00:18:14.603 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-03-09T00:18:14.688 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-03-09T00:18:14.699 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-03-09T00:18:14.753 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-03-09T00:18:14.795 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-03-09T00:18:14.880 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-03-09T00:18:14.891 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-03-09T00:18:14.966 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-03-09T00:18:14.998 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-03-09T00:18:15.072 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-03-09T00:18:15.105 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-03-09T00:18:15.169 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-03-09T00:18:15.191 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-03-09T00:18:15.266 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-03-09T00:18:15.303 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-03-09T00:18:15.367 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-03-09T00:18:15.400 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-03-09T00:18:15.469 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-03-09T00:18:15.491 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-03-09T00:18:15.586 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-03-09T00:18:15.587 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-03-09T00:18:15.666 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-03-09T00:18:15.688 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-03-09T00:18:15.784 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-03-09T00:18:15.796 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-03-09T00:18:15.861 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-03-09T00:18:15.883 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-03-09T00:18:15.988 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-03-09T00:18:16.000 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-03-09T00:18:16.056 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-03-09T00:18:16.078 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-03-09T00:18:16.174 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-03-09T00:18:16.196 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-03-09T00:18:16.249 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-03-09T00:18:16.271 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-03-09T00:18:16.366 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-03-09T00:18:16.409 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-03-09T00:18:16.453 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-03-09T00:18:16.475 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-03-09T00:18:16.570 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-03-09T00:18:16.636 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-03-09T00:18:16.638 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-03-09T00:18:16.702 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-03-09T00:18:16.776 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-03-09T00:18:16.839 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-03-09T00:18:16.851 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-03-09T00:18:16.894 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-03-09T00:18:16.968 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-03-09T00:18:17.032 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-03-09T00:18:17.054 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-03-09T00:18:17.087 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-03-09T00:18:17.172 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-03-09T00:18:17.227 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-03-09T00:18:17.262 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-03-09T00:18:17.284 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-03-09T00:18:17.369 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-03-09T00:18:17.422 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-03-09T00:18:17.466 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-03-09T00:18:17.478 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-03-09T00:18:17.563 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-03-09T00:18:17.617 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-03-09T00:18:17.681 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-03-09T00:18:17.682 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-03-09T00:18:17.756 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-03-09T00:18:17.811 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-03-09T00:18:17.874 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:17 +0000 (0:00:06.325) 0:01:09.847 ********** 2020-03-09T00:18:17.881 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-03-09T00:18:17.895 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-03-09T00:18:17.956 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-03-09T00:18:18.014 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.140) 0:01:09.987 ********** 2020-03-09T00:18:18.083 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-03-09T00:18:18.115 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-03-09T00:18:18.153 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.138) 0:01:10.126 ********** 2020-03-09T00:18:18.307 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-03-09T00:18:18.423 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-09T00:18:18.425 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "checksum": "ab8fb702ec4c081d6948785b08a4351b18f72652", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "01a29d289202b991c9cf8567348de928", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713097.9-229916698911917/source", "state": "file", "uid": 0} 2020-03-09T00:18:18.464 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.310) 0:01:10.437 ********** 2020-03-09T00:18:18.500 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-09T00:18:18.502 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => { "msg": "Host smithi104.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-09T00:18:18.544 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.079) 0:01:10.516 ********** 2020-03-09T00:18:18.556 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:18.557 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-03-09T00:18:18.559 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "checksum": "ab8fb702ec4c081d6948785b08a4351b18f72652", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "01a29d289202b991c9cf8567348de928", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713098.04-169843787508361/source", "state": "file", "uid": 0} 2020-03-09T00:18:18.602 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.057) 0:01:10.574 ********** 2020-03-09T00:18:18.610 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-09T00:18:18.611 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-09T00:18:18.653 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.051) 0:01:10.626 ********** 2020-03-09T00:18:18.659 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-09T00:18:18.660 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => { "msg": "Host smithi139.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-09T00:18:18.698 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.045) 0:01:10.671 ********** 2020-03-09T00:18:18.704 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-09T00:18:18.704 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "checksum": "ab8fb702ec4c081d6948785b08a4351b18f72652", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "01a29d289202b991c9cf8567348de928", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713098.18-178601155681804/source", "state": "file", "uid": 0} 2020-03-09T00:18:18.706 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:18.748 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.049) 0:01:10.720 ********** 2020-03-09T00:18:18.789 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.041) 0:01:10.762 ********** 2020-03-09T00:18:18.795 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:18.795 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-03-09T00:18:18.797 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-09T00:18:18.838 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.049) 0:01:10.811 ********** 2020-03-09T00:18:18.844 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-09T00:18:18.845 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => { "msg": "Host smithi148.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-09T00:18:18.882 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.043) 0:01:10.854 ********** 2020-03-09T00:18:18.887 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-09T00:18:18.888 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:18.929 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.047) 0:01:10.901 ********** 2020-03-09T00:18:18.934 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-09T00:18:18.935 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-09T00:18:18.973 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:18 +0000 (0:00:00.043) 0:01:10.945 ********** 2020-03-09T00:18:18.978 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:18.979 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-03-09T00:18:19.002 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-09T00:18:19.003 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.042 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.069) 0:01:11.015 ********** 2020-03-09T00:18:19.048 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-09T00:18:19.049 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.083212", "end": "2020-03-09 00:18:19.010660", "rc": 0, "start": "2020-03-09 00:18:18.927448", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-09T00:18:19.088 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.045) 0:01:11.061 ********** 2020-03-09T00:18:19.115 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-09T00:18:19.117 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.160 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.071) 0:01:11.133 ********** 2020-03-09T00:18:19.167 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:19.167 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-03-09T00:18:19.190 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-09T00:18:19.191 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.231 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.070) 0:01:11.203 ********** 2020-03-09T00:18:19.237 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-09T00:18:19.238 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.081710", "end": "2020-03-09 00:18:19.193546", "rc": 0, "start": "2020-03-09 00:18:19.111836", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-09T00:18:19.276 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.045) 0:01:11.249 ********** 2020-03-09T00:18:19.282 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-09T00:18:19.284 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.324 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.047) 0:01:11.296 ********** 2020-03-09T00:18:19.328 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-09T00:18:19.330 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-09T00:18:19.330 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.080877", "end": "2020-03-09 00:18:19.299952", "rc": 0, "start": "2020-03-09 00:18:19.219075", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-09T00:18:19.374 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.050) 0:01:11.346 ********** 2020-03-09T00:18:19.407 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.033) 0:01:11.380 ********** 2020-03-09T00:18:19.412 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-09T00:18:19.414 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi050.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-03-09T00:18:19.415 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.452 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.045) 0:01:11.425 ********** 2020-03-09T00:18:19.487 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.034) 0:01:11.460 ********** 2020-03-09T00:18:19.493 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-09T00:18:19.495 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.533 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.045) 0:01:11.506 ********** 2020-03-09T00:18:19.570 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.037) 0:01:11.543 ********** 2020-03-09T00:18:19.577 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-09T00:18:19.577 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.579 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi050.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-03-09T00:18:19.616 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.046) 0:01:11.589 ********** 2020-03-09T00:18:19.652 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.036) 0:01:11.625 ********** 2020-03-09T00:18:19.657 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-09T00:18:19.659 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.660 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-09T00:18:19.662 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi104.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:19.704 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.051) 0:01:11.676 ********** 2020-03-09T00:18:19.738 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.033) 0:01:11.710 ********** 2020-03-09T00:18:19.744 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-09T00:18:19.744 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.746 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.786 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.048) 0:01:11.759 ********** 2020-03-09T00:18:19.825 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.039) 0:01:11.798 ********** 2020-03-09T00:18:19.830 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-09T00:18:19.832 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi050.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-03-09T00:18:19.833 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:19.870 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.044) 0:01:11.843 ********** 2020-03-09T00:18:19.902 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.032) 0:01:11.875 ********** 2020-03-09T00:18:19.907 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-09T00:18:19.908 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi104.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:19.946 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.043) 0:01:11.919 ********** 2020-03-09T00:18:19.986 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:19 +0000 (0:00:00.039) 0:01:11.958 ********** 2020-03-09T00:18:19.992 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-09T00:18:19.993 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.030 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.044) 0:01:12.003 ********** 2020-03-09T00:18:20.072 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.041) 0:01:12.045 ********** 2020-03-09T00:18:20.078 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:20.079 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-03-09T00:18:20.080 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.082 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi104.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:20.118 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.045) 0:01:12.090 ********** 2020-03-09T00:18:20.157 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.038) 0:01:12.129 ********** 2020-03-09T00:18:20.162 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-09T00:18:20.164 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.206 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.049) 0:01:12.178 ********** 2020-03-09T00:18:20.242 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.035) 0:01:12.214 ********** 2020-03-09T00:18:20.247 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-09T00:18:20.248 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:20.249 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Remove revoked users] ******************************************** 2020-03-09T00:18:20.250 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-03-09T00:18:20.251 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.288 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.046) 0:01:12.261 ********** 2020-03-09T00:18:20.328 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.039) 0:01:12.300 ********** 2020-03-09T00:18:20.334 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-09T00:18:20.336 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.377 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.049) 0:01:12.350 ********** 2020-03-09T00:18:20.414 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.036) 0:01:12.386 ********** 2020-03-09T00:18:20.419 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-09T00:18:20.423 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.424 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.426 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi050.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-03-09T00:18:20.482 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.068) 0:01:12.455 ********** 2020-03-09T00:18:20.514 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.032) 0:01:12.487 ********** 2020-03-09T00:18:20.549 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.034) 0:01:12.521 ********** 2020-03-09T00:18:20.554 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-09T00:18:20.555 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:20.556 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi104.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:20.597 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.048) 0:01:12.569 ********** 2020-03-09T00:18:20.631 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.033) 0:01:12.603 ********** 2020-03-09T00:18:20.635 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-09T00:18:20.637 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Clean yum cache] ************************************************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.678 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.047) 0:01:12.650 ********** 2020-03-09T00:18:20.716 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.038) 0:01:12.689 ********** 2020-03-09T00:18:20.722 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-09T00:18:20.722 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.724 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.765 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.048) 0:01:12.737 ********** 2020-03-09T00:18:20.797 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.031) 0:01:12.769 ********** 2020-03-09T00:18:20.802 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-09T00:18:20.802 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:20.804 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.842 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.045) 0:01:12.814 ********** 2020-03-09T00:18:20.876 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.034) 0:01:12.849 ********** 2020-03-09T00:18:20.881 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-09T00:18:20.883 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.921 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.044) 0:01:12.893 ********** 2020-03-09T00:18:20.954 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.033) 0:01:12.927 ********** 2020-03-09T00:18:20.960 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-09T00:18:20.960 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:20.962 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.004 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:20 +0000 (0:00:00.049) 0:01:12.976 ********** 2020-03-09T00:18:21.040 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.036) 0:01:13.013 ********** 2020-03-09T00:18:21.047 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-09T00:18:21.048 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:21.049 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-09T00:18:21.051 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "checksum": "ab8fb702ec4c081d6948785b08a4351b18f72652", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "01a29d289202b991c9cf8567348de928", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583713100.5-5018228084113/source", "state": "file", "uid": 0} 2020-03-09T00:18:21.052 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.089 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.049) 0:01:13.062 ********** 2020-03-09T00:18:21.125 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.035) 0:01:13.098 ********** 2020-03-09T00:18:21.160 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.034) 0:01:13.132 ********** 2020-03-09T00:18:21.167 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-09T00:18:21.168 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:21.169 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi050.front.sepia.ceph.com] => { "msg": "Host smithi050.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-09T00:18:21.171 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi104.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:21.210 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.050) 0:01:13.183 ********** 2020-03-09T00:18:21.245 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.034) 0:01:13.218 ********** 2020-03-09T00:18:21.287 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.041) 0:01:13.259 ********** 2020-03-09T00:18:21.293 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-09T00:18:21.294 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.296 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-09T00:18:21.298 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-09T00:18:21.299 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-09T00:18:21.338 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.051) 0:01:13.311 ********** 2020-03-09T00:18:21.376 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.037) 0:01:13.349 ********** 2020-03-09T00:18:21.412 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.036) 0:01:13.385 ********** 2020-03-09T00:18:21.417 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-09T00:18:21.418 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.420 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.421 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.460 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.047) 0:01:13.432 ********** 2020-03-09T00:18:21.494 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.033) 0:01:13.466 ********** 2020-03-09T00:18:21.528 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.034) 0:01:13.500 ********** 2020-03-09T00:18:21.535 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-09T00:18:21.536 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:21.577 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.049) 0:01:13.550 ********** 2020-03-09T00:18:21.582 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-09T00:18:21.583 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.624 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.047) 0:01:13.597 ********** 2020-03-09T00:18:21.634 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-09T00:18:21.635 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.675 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.050) 0:01:13.647 ********** 2020-03-09T00:18:21.679 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-09T00:18:21.681 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.723 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.048) 0:01:13.696 ********** 2020-03-09T00:18:21.729 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-09T00:18:21.729 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.731 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi104.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:21.772 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.048) 0:01:13.745 ********** 2020-03-09T00:18:21.809 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.033) 0:01:13.779 ********** 2020-03-09T00:18:21.815 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-09T00:18:21.816 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.817 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.067202", "end": "2020-03-09 00:18:21.731983", "rc": 0, "start": "2020-03-09 00:18:21.664781", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-09T00:18:21.855 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.048) 0:01:13.828 ********** 2020-03-09T00:18:21.890 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.034) 0:01:13.862 ********** 2020-03-09T00:18:21.895 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-09T00:18:21.897 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:21.933 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.043) 0:01:13.906 ********** 2020-03-09T00:18:21.969 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:21 +0000 (0:00:00.035) 0:01:13.941 ********** 2020-03-09T00:18:21.974 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-09T00:18:21.974 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:21.976 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.014 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.044) 0:01:13.986 ********** 2020-03-09T00:18:22.050 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.036) 0:01:14.023 ********** 2020-03-09T00:18:22.055 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-09T00:18:22.057 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.095 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.044) 0:01:14.067 ********** 2020-03-09T00:18:22.136 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.041) 0:01:14.109 ********** 2020-03-09T00:18:22.142 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-09T00:18:22.143 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.144 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.183 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.046) 0:01:14.156 ********** 2020-03-09T00:18:22.218 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.035) 0:01:14.191 ********** 2020-03-09T00:18:22.223 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-09T00:18:22.225 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.262 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.043) 0:01:14.235 ********** 2020-03-09T00:18:22.300 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.037) 0:01:14.272 ********** 2020-03-09T00:18:22.304 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-09T00:18:22.305 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.306 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.344 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.044) 0:01:14.317 ********** 2020-03-09T00:18:22.378 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.033) 0:01:14.351 ********** 2020-03-09T00:18:22.383 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-09T00:18:22.383 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.385 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi050.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:22.424 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.045) 0:01:14.397 ********** 2020-03-09T00:18:22.460 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.035) 0:01:14.432 ********** 2020-03-09T00:18:22.465 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-09T00:18:22.465 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.467 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.509 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.049) 0:01:14.482 ********** 2020-03-09T00:18:22.542 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.032) 0:01:14.515 ********** 2020-03-09T00:18:22.547 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-09T00:18:22.548 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.588 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.045) 0:01:14.560 ********** 2020-03-09T00:18:22.621 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.033) 0:01:14.594 ********** 2020-03-09T00:18:22.626 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-09T00:18:22.627 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-09T00:18:22.629 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.630 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:22.671 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.049) 0:01:14.643 ********** 2020-03-09T00:18:22.704 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.032) 0:01:14.676 ********** 2020-03-09T00:18:22.711 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-09T00:18:22.712 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.747 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.043) 0:01:14.720 ********** 2020-03-09T00:18:22.780 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.032) 0:01:14.752 ********** 2020-03-09T00:18:22.783 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-09T00:18:22.785 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Register with subscription-manager.] **************************** skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:22.831 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.051) 0:01:14.803 ********** 2020-03-09T00:18:22.864 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.033) 0:01:14.837 ********** 2020-03-09T00:18:22.870 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-09T00:18:22.872 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.909 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.044) 0:01:14.881 ********** 2020-03-09T00:18:22.944 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.035) 0:01:14.917 ********** 2020-03-09T00:18:22.949 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-09T00:18:22.950 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.951 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:22.994 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:22 +0000 (0:00:00.050) 0:01:14.967 ********** 2020-03-09T00:18:23.033 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.038) 0:01:15.006 ********** 2020-03-09T00:18:23.039 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-09T00:18:23.039 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.041 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-09T00:18:23.042 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-09T00:18:23.046 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-09T00:18:23.047 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.083 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.050) 0:01:15.056 ********** 2020-03-09T00:18:23.116 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.032) 0:01:15.089 ********** 2020-03-09T00:18:23.150 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.033) 0:01:15.123 ********** 2020-03-09T00:18:23.156 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-09T00:18:23.156 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:23.158 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.196 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.046) 0:01:15.169 ********** 2020-03-09T00:18:23.229 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.032) 0:01:15.202 ********** 2020-03-09T00:18:23.235 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-09T00:18:23.237 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:23.276 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.046) 0:01:15.248 ********** 2020-03-09T00:18:23.311 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.035) 0:01:15.284 ********** 2020-03-09T00:18:23.315 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-09T00:18:23.316 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-09T00:18:23.317 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.355 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.043) 0:01:15.327 ********** 2020-03-09T00:18:23.390 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.035) 0:01:15.363 ********** 2020-03-09T00:18:23.396 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-09T00:18:23.397 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:23.398 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.435 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.045) 0:01:15.408 ********** 2020-03-09T00:18:23.470 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.034) 0:01:15.443 ********** 2020-03-09T00:18:23.476 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-09T00:18:23.478 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Store list of enabled repos] ************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.514 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.043) 0:01:15.487 ********** 2020-03-09T00:18:23.549 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.034) 0:01:15.521 ********** 2020-03-09T00:18:23.553 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-09T00:18:23.554 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-09T00:18:23.554 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.599 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.050) 0:01:15.572 ********** 2020-03-09T00:18:23.634 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.034) 0:01:15.606 ********** 2020-03-09T00:18:23.639 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-09T00:18:23.640 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-09T00:18:23.642 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.683 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.049) 0:01:15.656 ********** 2020-03-09T00:18:23.689 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-09T00:18:23.691 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.727 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.044) 0:01:15.700 ********** 2020-03-09T00:18:23.735 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-09T00:18:23.736 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:23.777 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.049) 0:01:15.749 ********** 2020-03-09T00:18:23.783 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-09T00:18:23.784 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:23.821 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.044) 0:01:15.794 ********** 2020-03-09T00:18:23.826 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-09T00:18:23.827 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-09T00:18:23.865 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.043) 0:01:15.837 ********** 2020-03-09T00:18:23.893 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-09T00:18:23.929 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.064) 0:01:15.902 ********** 2020-03-09T00:18:23.945 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-09T00:18:23.947 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:23.983 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:23 +0000 (0:00:00.053) 0:01:15.955 ********** 2020-03-09T00:18:23.988 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-09T00:18:23.990 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.026 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.043) 0:01:15.999 ********** 2020-03-09T00:18:24.096 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-09T00:18:24.098 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-09T00:18:24.135 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.108) 0:01:16.108 ********** 2020-03-09T00:18:24.163 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-09T00:18:24.164 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.202 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.067) 0:01:16.175 ********** 2020-03-09T00:18:24.229 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-09T00:18:24.230 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.267 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.064) 0:01:16.239 ********** 2020-03-09T00:18:24.269 INFO:teuthology.task.ansible.out: 2020-03-09T00:18:24.272 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-09T00:18:24.273 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:24.319 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.051) 0:01:16.291 ********** 2020-03-09T00:18:24.325 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-09T00:18:24.327 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.364 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.045) 0:01:16.337 ********** 2020-03-09T00:18:24.370 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-09T00:18:24.372 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-09T00:18:24.394 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-09T00:18:24.396 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.433 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.068) 0:01:16.406 ********** 2020-03-09T00:18:24.460 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-09T00:18:24.462 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.497 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.063) 0:01:16.470 ********** 2020-03-09T00:18:24.514 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-09T00:18:24.515 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-09T00:18:24.529 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-09T00:18:24.531 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.568 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.071) 0:01:16.541 ********** 2020-03-09T00:18:24.598 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-09T00:18:24.600 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.648 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.079) 0:01:16.621 ********** 2020-03-09T00:18:24.675 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-09T00:18:24.677 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.721 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.072) 0:01:16.694 ********** 2020-03-09T00:18:24.751 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-09T00:18:24.753 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:24.789 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.067) 0:01:16.761 ********** 2020-03-09T00:18:24.796 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-09T00:18:24.797 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-09T00:18:24.841 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:24 +0000 (0:00:00.052) 0:01:16.813 ********** 2020-03-09T00:18:24.860 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-09T00:18:24.970 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-09T00:18:25.021 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:25 +0000 (0:00:00.180) 0:01:16.994 ********** 2020-03-09T00:18:25.334 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-09T00:18:25.334 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-09T00:18:25.346 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-09T00:18:25.347 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-09T00:18:25.390 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:25 +0000 (0:00:00.369) 0:01:17.363 ********** 2020-03-09T00:18:25.514 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-09T00:18:25.515 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-09T00:18:25.526 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-09T00:18:25.527 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.538006", "end": "2020-03-09 00:18:25.500317", "rc": 0, "start": "2020-03-09 00:18:24.962311", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:18:25.568 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:25 +0000 (0:00:00.177) 0:01:17.540 ********** 2020-03-09T00:18:25.806 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-09T00:18:25.807 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-09T00:18:25.848 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:25 +0000 (0:00:00.280) 0:01:17.821 ********** 2020-03-09T00:18:25.877 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-09T00:18:25.879 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:25.915 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:25 +0000 (0:00:00.066) 0:01:17.887 ********** 2020-03-09T00:18:25.921 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-09T00:18:25.925 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-09T00:18:25.944 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-09T00:18:25.946 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.000 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:25 +0000 (0:00:00.079) 0:01:17.967 ********** 2020-03-09T00:18:26.002 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi148.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-09T00:18:26.041 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.046) 0:01:18.014 ********** 2020-03-09T00:18:26.046 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-09T00:18:26.047 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.087 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.046) 0:01:18.060 ********** 2020-03-09T00:18:26.093 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-09T00:18:26.094 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.132 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.045) 0:01:18.105 ********** 2020-03-09T00:18:26.137 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-09T00:18:26.139 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.178 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.045) 0:01:18.150 ********** 2020-03-09T00:18:26.184 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-09T00:18:26.185 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.227 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.048) 0:01:18.199 ********** 2020-03-09T00:18:26.233 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-09T00:18:26.236 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.276 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.049) 0:01:18.249 ********** 2020-03-09T00:18:26.284 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-09T00:18:26.285 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.334 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.057) 0:01:18.306 ********** 2020-03-09T00:18:26.340 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-09T00:18:26.342 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.384 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.049) 0:01:18.356 ********** 2020-03-09T00:18:26.390 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-09T00:18:26.392 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.436 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.052) 0:01:18.409 ********** 2020-03-09T00:18:26.441 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-09T00:18:26.442 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-09T00:18:26.443 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-09T00:18:26.484 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.047) 0:01:18.456 ********** 2020-03-09T00:18:26.522 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.038) 0:01:18.495 ********** 2020-03-09T00:18:26.529 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-09T00:18:26.530 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.531 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.569 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.046) 0:01:18.542 ********** 2020-03-09T00:18:26.602 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.032) 0:01:18.575 ********** 2020-03-09T00:18:26.606 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-09T00:18:26.607 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.609 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.646 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.043) 0:01:18.618 ********** 2020-03-09T00:18:26.683 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.037) 0:01:18.655 ********** 2020-03-09T00:18:26.688 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-09T00:18:26.689 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.732 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.049) 0:01:18.705 ********** 2020-03-09T00:18:26.774 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.041) 0:01:18.746 ********** 2020-03-09T00:18:26.779 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-09T00:18:26.780 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.781 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-09T00:18:26.783 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-09T00:18:26.823 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.049) 0:01:18.795 ********** 2020-03-09T00:18:26.858 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.034) 0:01:18.830 ********** 2020-03-09T00:18:26.889 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.031) 0:01:18.862 ********** 2020-03-09T00:18:26.894 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-09T00:18:26.895 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Get the current timezone] *************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Clean apt cache] ************************************************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:26.932 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.042) 0:01:18.905 ********** 2020-03-09T00:18:26.967 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.034) 0:01:18.939 ********** 2020-03-09T00:18:26.999 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:26 +0000 (0:00:00.032) 0:01:18.972 ********** 2020-03-09T00:18:27.003 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-09T00:18:27.004 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.005 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.042 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.043) 0:01:19.015 ********** 2020-03-09T00:18:27.074 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.030) 0:01:19.046 ********** 2020-03-09T00:18:27.079 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-09T00:18:27.080 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.081 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.116 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.042) 0:01:19.089 ********** 2020-03-09T00:18:27.148 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.031) 0:01:19.120 ********** 2020-03-09T00:18:27.152 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-09T00:18:27.153 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.157 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.191 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.042) 0:01:19.163 ********** 2020-03-09T00:18:27.222 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.031) 0:01:19.195 ********** 2020-03-09T00:18:27.228 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-09T00:18:27.229 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.267 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.044) 0:01:19.239 ********** 2020-03-09T00:18:27.300 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.033) 0:01:19.273 ********** 2020-03-09T00:18:27.306 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-09T00:18:27.307 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:27.345 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.044) 0:01:19.318 ********** 2020-03-09T00:18:27.524 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.018916", "end": "2020-03-09 00:18:27.500328", "rc": 0, "start": "2020-03-09 00:18:27.481412", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.433749", "end": "2020-03-09 00:18:27.500326", "rc": 0, "start": "2020-03-09 00:18:27.066577", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.045161", "end": "2020-03-09 00:18:27.500334", "rc": 0, "start": "2020-03-09 00:18:27.455173", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:18:27.563 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.217) 0:01:19.536 ********** 2020-03-09T00:18:27.602 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.037) 0:01:19.573 ********** 2020-03-09T00:18:27.635 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.033) 0:01:19.607 ********** 2020-03-09T00:18:27.830 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-09T00:18:27.831 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-09T00:18:27.870 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:27 +0000 (0:00:00.235) 0:01:19.843 ********** 2020-03-09T00:18:28.528 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-09T00:18:28.529 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-09T00:18:28.571 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:28 +0000 (0:00:00.700) 0:01:20.543 ********** 2020-03-09T00:18:28.599 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-09T00:18:28.600 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:28.637 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:28 +0000 (0:00:00.066) 0:01:20.610 ********** 2020-03-09T00:18:28.665 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-09T00:18:28.667 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:28.705 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:28 +0000 (0:00:00.067) 0:01:20.678 ********** 2020-03-09T00:18:28.732 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-09T00:18:28.733 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:28.777 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:28 +0000 (0:00:00.071) 0:01:20.749 ********** 2020-03-09T00:18:28.783 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-09T00:18:28.784 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-09T00:18:28.825 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:28 +0000 (0:00:00.048) 0:01:20.797 ********** 2020-03-09T00:18:28.831 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-09T00:18:28.831 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:28.871 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:28 +0000 (0:00:00.045) 0:01:20.843 ********** 2020-03-09T00:18:29.339 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:29.340 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-09T00:18:29.362 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-09T00:18:29.364 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-09T00:18:29.405 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:29 +0000 (0:00:00.534) 0:01:21.378 ********** 2020-03-09T00:18:29.807 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:29.808 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-09T00:18:30.263 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-09T00:18:30.304 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.899) 0:01:22.277 ********** 2020-03-09T00:18:30.369 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-09T00:18:30.402 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.097) 0:01:22.375 ********** 2020-03-09T00:18:30.408 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-09T00:18:30.409 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-09T00:18:30.448 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.046) 0:01:22.421 ********** 2020-03-09T00:18:30.454 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-09T00:18:30.498 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.049) 0:01:22.470 ********** 2020-03-09T00:18:30.502 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-09T00:18:30.503 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.505 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:18:30.543 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.045) 0:01:22.516 ********** 2020-03-09T00:18:30.577 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.033) 0:01:22.549 ********** 2020-03-09T00:18:30.582 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-09T00:18:30.583 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.621 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.044) 0:01:22.593 ********** 2020-03-09T00:18:30.626 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-09T00:18:30.628 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.665 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.044) 0:01:22.638 ********** 2020-03-09T00:18:30.674 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.710 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.044) 0:01:22.682 ********** 2020-03-09T00:18:30.716 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-09T00:18:30.717 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.767 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.057) 0:01:22.740 ********** 2020-03-09T00:18:30.773 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.815 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.047) 0:01:22.788 ********** 2020-03-09T00:18:30.821 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-09T00:18:30.822 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.865 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.049) 0:01:22.838 ********** 2020-03-09T00:18:30.874 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:30.916 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:30 +0000 (0:00:00.051) 0:01:22.889 ********** 2020-03-09T00:18:30.922 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:30.923 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-09T00:18:30.999 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-09T00:18:31.001 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-09T00:18:31.347 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-09T00:18:31.384 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:31 +0000 (0:00:00.467) 0:01:23.356 ********** 2020-03-09T00:18:31.391 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-09T00:18:31.433 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:31 +0000 (0:00:00.046) 0:01:23.403 ********** 2020-03-09T00:18:31.439 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:31.440 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-09T00:18:31.452 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-09T00:18:31.453 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-09T00:18:31.896 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-09T00:18:31.900 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:31.901 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-09T00:18:31.944 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:31 +0000 (0:00:00.514) 0:01:23.917 ********** 2020-03-09T00:18:31.983 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:31 +0000 (0:00:00.038) 0:01:23.956 ********** 2020-03-09T00:18:32.013 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-09T00:18:32.015 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:32.049 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:32 +0000 (0:00:00.066) 0:01:24.022 ********** 2020-03-09T00:18:32.086 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-09T00:18:32.088 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-09T00:18:32.124 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:32 +0000 (0:00:00.074) 0:01:24.097 ********** 2020-03-09T00:18:32.151 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-09T00:18:32.152 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:18:32.189 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:32 +0000 (0:00:00.064) 0:01:24.161 ********** 2020-03-09T00:18:32.312 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-09T00:18:32.314 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-09T00:18:32.355 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:32 +0000 (0:00:00.165) 0:01:24.327 ********** 2020-03-09T00:18:32.360 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-09T00:18:32.361 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-09T00:18:32.362 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-09T00:18:32.401 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:32 +0000 (0:00:00.046) 0:01:24.374 ********** 2020-03-09T00:18:32.434 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:32 +0000 (0:00:00.032) 0:01:24.406 ********** 2020-03-09T00:18:32.440 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-09T00:18:32.440 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:32.452 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:32.463 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:32.477 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:32.515 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:32 +0000 (0:00:00.081) 0:01:24.487 ********** 2020-03-09T00:18:32.679 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-09T00:18:32.680 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-09T00:18:32.853 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:32.855 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-09T00:18:32.927 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-09T00:18:33.148 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-09T00:18:33.150 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-09T00:18:33.307 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:33.309 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-09T00:18:33.374 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-09T00:18:33.463 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-09T00:18:33.464 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-09T00:18:33.519 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:33 +0000 (0:00:01.004) 0:01:25.492 ********** 2020-03-09T00:18:33.547 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-09T00:18:33.549 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:33.587 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:33 +0000 (0:00:00.067) 0:01:25.559 ********** 2020-03-09T00:18:33.614 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-09T00:18:33.616 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-09T00:18:33.655 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:33 +0000 (0:00:00.068) 0:01:25.628 ********** 2020-03-09T00:18:33.682 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-09T00:18:33.683 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:33.723 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:33 +0000 (0:00:00.068) 0:01:25.696 ********** 2020-03-09T00:18:33.751 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:33.752 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-09T00:18:33.755 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-09T00:18:33.791 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:33 +0000 (0:00:00.068) 0:01:25.764 ********** 2020-03-09T00:18:33.827 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:33 +0000 (0:00:00.035) 0:01:25.799 ********** 2020-03-09T00:18:33.832 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-09T00:18:33.833 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-09T00:18:33.875 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:33 +0000 (0:00:00.047) 0:01:25.847 ********** 2020-03-09T00:18:34.007 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-09T00:18:34.009 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-09T00:18:34.052 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.177) 0:01:26.025 ********** 2020-03-09T00:18:34.063 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-09T00:18:34.064 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:18:34.065 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-09T00:18:34.066 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-09T00:18:34.068 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:18:34.107 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.054) 0:01:26.079 ********** 2020-03-09T00:18:34.146 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.039) 0:01:26.118 ********** 2020-03-09T00:18:34.187 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.041) 0:01:26.160 ********** 2020-03-09T00:18:34.193 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-09T00:18:34.195 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:34.196 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:34.234 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.047) 0:01:26.207 ********** 2020-03-09T00:18:34.586 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-09T00:18:34.586 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-09T00:18:34.632 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.398) 0:01:26.605 ********** 2020-03-09T00:18:34.640 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi139.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-09T00:18:34.651 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-09T00:18:34.663 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-09T00:18:34.664 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:34.707 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.075) 0:01:26.680 ********** 2020-03-09T00:18:34.930 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi104.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:34.979 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:34 +0000 (0:00:00.271) 0:01:26.951 ********** 2020-03-09T00:18:35.090 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-09T00:18:35.102 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-09T00:18:35.104 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-09T00:18:35.224 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-09T00:18:35.226 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi104.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-09T00:18:35.262 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.283) 0:01:27.235 ********** 2020-03-09T00:18:35.321 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.058) 0:01:27.294 ********** 2020-03-09T00:18:35.327 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-09T00:18:35.329 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:35.376 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.054) 0:01:27.348 ********** 2020-03-09T00:18:35.384 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-09T00:18:35.385 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:35.424 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.048) 0:01:27.397 ********** 2020-03-09T00:18:35.538 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-09T00:18:35.540 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-09T00:18:35.592 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.167) 0:01:27.564 ********** 2020-03-09T00:18:35.602 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-09T00:18:35.645 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.052) 0:01:27.617 ********** 2020-03-09T00:18:35.651 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-09T00:18:35.652 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:35.693 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.047) 0:01:27.665 ********** 2020-03-09T00:18:35.699 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:35.739 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.046) 0:01:27.712 ********** 2020-03-09T00:18:35.744 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-09T00:18:35.745 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-09T00:18:35.783 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.043) 0:01:27.755 ********** 2020-03-09T00:18:35.788 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-09T00:18:35.828 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.043) 0:01:27.799 ********** 2020-03-09T00:18:35.873 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-09T00:18:35.875 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-09T00:18:35.909 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:35 +0000 (0:00:00.083) 0:01:27.882 ********** 2020-03-09T00:18:35.979 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-09T00:18:35.982 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-09T00:18:36.017 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.107) 0:01:27.989 ********** 2020-03-09T00:18:36.033 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-09T00:18:36.072 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.054) 0:01:28.044 ********** 2020-03-09T00:18:36.078 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-09T00:18:36.079 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.080 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.091 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.133 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.061) 0:01:28.105 ********** 2020-03-09T00:18:36.139 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.140 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.142 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.153 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.190 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.057) 0:01:28.163 ********** 2020-03-09T00:18:36.408 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-09T00:18:36.450 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.259) 0:01:28.422 ********** 2020-03-09T00:18:36.477 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-09T00:18:36.478 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:36.531 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.081) 0:01:28.504 ********** 2020-03-09T00:18:36.751 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-09T00:18:36.753 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:36.795 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.263) 0:01:28.767 ********** 2020-03-09T00:18:36.801 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-09T00:18:36.802 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-09T00:18:36.842 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:36 +0000 (0:00:00.047) 0:01:28.815 ********** 2020-03-09T00:18:37.048 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-09T00:18:37.053 INFO:teuthology.task.ansible.out:fatal: [smithi050.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-09T00:18:37.087 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.244) 0:01:29.059 ********** 2020-03-09T00:18:37.094 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-09T00:18:37.095 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-09T00:18:37.134 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.047) 0:01:29.107 ********** 2020-03-09T00:18:37.140 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-09T00:18:37.141 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:37.143 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-09T00:18:37.181 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.046) 0:01:29.153 ********** 2020-03-09T00:18:37.215 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.034) 0:01:29.188 ********** 2020-03-09T00:18:37.221 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-09T00:18:37.223 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:37.260 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.044) 0:01:29.232 ********** 2020-03-09T00:18:37.265 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:37.303 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.043) 0:01:29.276 ********** 2020-03-09T00:18:37.665 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-09T00:18:37.668 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:13:35 UTC", "ActiveEnterTimestampMonotonic": "14790855", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount -.mount tmp.mount system.slice basic.target time-sync.target systemd-journald.socket systemd-tmpfiles-setup.service local-fs.target sysinit.target network.target nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:13:35 UTC", "AssertTimestampMonotonic": "14789942", "Before": "xdm.service getty@tty1.service multi-user.target plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:13:35 UTC", "ConditionTimestampMonotonic": "14789942", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-09T00:18:37.669 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2408", "ExecMainStartTimestamp": "Mon 2020-03-09 00:13:35 UTC", "ExecMainStartTimestampMonotonic": "14790831", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Mon 2020-03-09 00:13:35 UTC] ; stop_time=[n/a] ; pid=2408 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:13:35 UTC", "InactiveExitTimestampMonotonic": "14790855", "InvocationID": "a773cdc7ad2b4398beab9b39c0593437", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-09T00:18:37.671 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127443", "LimitNPROCSoft": "127443", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127443", "LimitSIGPENDINGSoft": "127443", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2408", "MemoryAccounting": "yes", "MemoryCurrent": "1384448", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-09T00:18:37.673 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:13:35 UTC", "StateChangeTimestampMonotonic": "14790855", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:13:35 UTC", "WatchdogTimestampMonotonic": "14790854", "WatchdogUSec": "0"}} 2020-03-09T00:18:37.715 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.412) 0:01:29.688 ********** 2020-03-09T00:18:37.724 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-09T00:18:37.725 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-09T00:18:37.765 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.049) 0:01:29.738 ********** 2020-03-09T00:18:37.805 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.039) 0:01:29.777 ********** 2020-03-09T00:18:37.810 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-09T00:18:37.848 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.043) 0:01:29.821 ********** 2020-03-09T00:18:37.907 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-09T00:18:37.908 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001721", "end": "2020-03-09 00:18:37.884649", "rc": 0, "start": "2020-03-09 00:18:37.882928", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-09T00:18:37.946 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:37 +0000 (0:00:00.097) 0:01:29.919 ********** 2020-03-09T00:18:38.100 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-09T00:18:38.102 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:13:07 UTC", "ActiveEnterTimestampMonotonic": "17814352", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target -.mount systemd-journald.socket network.target systemd-tmpfiles-setup.service sysinit.target tmp.mount nss-lookup.target time-sync.target var-run.mount local-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:13:07 UTC", "AssertTimestampMonotonic": "17813476", "Before": "multi-user.target plymouth-quit.service getty@tty1.service xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:13:07 UTC", "ConditionTimestampMonotonic": "17813476", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-09T00:18:38.104 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2423", "ExecMainStartTimestamp": "Mon 2020-03-09 00:13:07 UTC", "ExecMainStartTimestampMonotonic": "17814320", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Mon 2020-03-09 00:13:07 UTC] ; stop_time=[n/a] ; pid=2423 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:13:07 UTC", "InactiveExitTimestampMonotonic": "17814352", "InvocationID": "f1c30415ad6a428d955b5028d0441931", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-09T00:18:38.105 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127467", "LimitNPROCSoft": "127467", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127467", "LimitSIGPENDINGSoft": "127467", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2423", "MemoryAccounting": "yes", "MemoryCurrent": "1417216", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-09T00:18:38.107 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:13:07 UTC", "StateChangeTimestampMonotonic": "17814352", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:13:07 UTC", "WatchdogTimestampMonotonic": "17814351", "WatchdogUSec": "0"}} 2020-03-09T00:18:38.142 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.196) 0:01:30.115 ********** 2020-03-09T00:18:38.284 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-09T00:18:38.285 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-09T00:18:38.322 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.180) 0:01:30.295 ********** 2020-03-09T00:18:38.339 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-09T00:18:38.375 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.052) 0:01:30.348 ********** 2020-03-09T00:18:38.383 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-09T00:18:38.384 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001790", "end": "2020-03-09 00:18:38.326874", "rc": 0, "start": "2020-03-09 00:18:38.325084", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-09T00:18:38.385 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:38.421 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.045) 0:01:30.394 ********** 2020-03-09T00:18:38.456 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.035) 0:01:30.429 ********** 2020-03-09T00:18:38.462 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:38.501 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.044) 0:01:30.473 ********** 2020-03-09T00:18:38.663 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-09T00:18:38.664 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:38.702 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.201) 0:01:30.674 ********** 2020-03-09T00:18:38.719 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:18:38.757 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.054) 0:01:30.729 ********** 2020-03-09T00:18:38.848 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-09T00:18:38.849 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-09T00:18:38.886 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.129) 0:01:30.859 ********** 2020-03-09T00:18:38.955 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-09T00:18:38.957 INFO:teuthology.task.ansible.out:fatal: [smithi148.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-09T00:18:38.994 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:38 +0000 (0:00:00.107) 0:01:30.966 ********** 2020-03-09T00:18:39.021 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-09T00:18:39.021 INFO:teuthology.task.ansible.out:...ignoring 2020-03-09T00:18:39.023 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:39.061 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.067) 0:01:31.034 ********** 2020-03-09T00:18:39.096 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.034) 0:01:31.069 ********** 2020-03-09T00:18:39.101 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:18:39.140 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.044) 0:01:31.113 ********** 2020-03-09T00:18:39.400 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-09T00:18:39.436 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.295) 0:01:31.409 ********** 2020-03-09T00:18:39.570 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-09T00:18:39.611 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.174) 0:01:31.583 ********** 2020-03-09T00:18:39.616 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-09T00:18:39.656 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.045) 0:01:31.629 ********** 2020-03-09T00:18:39.790 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-09T00:18:39.827 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.171) 0:01:31.800 ********** 2020-03-09T00:18:39.930 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-09T00:18:39.933 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:16:37 UTC", "ActiveEnterTimestampMonotonic": "11572543", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target sysinit.target -.mount systemd-tmpfiles-setup.service time-sync.target system.slice systemd-journald.socket network.target tmp.mount local-fs.target var-run.mount nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:16:37 UTC", "AssertTimestampMonotonic": "11571678", "Before": "shutdown.target multi-user.target xdm.service plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:16:37 UTC", "ConditionTimestampMonotonic": "11571678", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-09T00:18:39.935 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2393", "ExecMainStartTimestamp": "Mon 2020-03-09 00:16:37 UTC", "ExecMainStartTimestampMonotonic": "11572510", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Mon 2020-03-09 00:16:37 UTC] ; stop_time=[n/a] ; pid=2393 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:16:37 UTC", "InactiveExitTimestampMonotonic": "11572543", "InvocationID": "733f19c888274165afedf4cd58f986ce", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-09T00:18:39.938 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127437", "LimitNPROCSoft": "127437", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127437", "LimitSIGPENDINGSoft": "127437", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2393", "MemoryAccounting": "yes", "MemoryCurrent": "1564672", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-09T00:18:39.940 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:16:37 UTC", "StateChangeTimestampMonotonic": "11572543", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:16:37 UTC", "WatchdogTimestampMonotonic": "11572542", "WatchdogUSec": "0"}} 2020-03-09T00:18:39.988 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:39 +0000 (0:00:00.160) 0:01:31.961 ********** 2020-03-09T00:18:39.999 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:16:40 UTC", "ActiveEnterTimestampMonotonic": "12506874", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount time-sync.target var-run.mount system.slice local-fs.target tmp.mount network.target sysinit.target systemd-tmpfiles-setup.service nss-lookup.target basic.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:16:40 UTC", "AssertTimestampMonotonic": "12506100", "Before": "shutdown.target plymouth-quit.service getty@tty1.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:16:40 UTC", "ConditionTimestampMonotonic": "12506100", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-09T00:18:40.001 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2391", "ExecMainStartTimestamp": "Mon 2020-03-09 00:16:40 UTC", "ExecMainStartTimestampMonotonic": "12506844", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Mon 2020-03-09 00:16:40 UTC] ; stop_time=[n/a] ; pid=2391 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:16:40 UTC", "InactiveExitTimestampMonotonic": "12506874", "InvocationID": "241d73330b384d08b6b3f3bf7446760c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-09T00:18:40.003 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2391", "MemoryAccounting": "yes", "MemoryCurrent": "1421312", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-09T00:18:40.004 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:16:40 UTC", "StateChangeTimestampMonotonic": "12506874", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:16:40 UTC", "WatchdogTimestampMonotonic": "12506873", "WatchdogUSec": "0"}} 2020-03-09T00:18:40.046 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:40 +0000 (0:00:00.057) 0:01:32.019 ********** 2020-03-09T00:18:40.189 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-09T00:18:40.190 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001771", "end": "2020-03-09 00:18:40.163597", "rc": 0, "start": "2020-03-09 00:18:40.161826", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-09T00:18:40.228 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:40 +0000 (0:00:00.181) 0:01:32.200 ********** 2020-03-09T00:18:40.244 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001816", "end": "2020-03-09 00:18:40.219312", "rc": 0, "start": "2020-03-09 00:18:40.217496", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-09T00:18:40.288 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:40 +0000 (0:00:00.059) 0:01:32.260 ********** 2020-03-09T00:18:40.447 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-09T00:18:40.448 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-09T00:18:40.493 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:40 +0000 (0:00:00.205) 0:01:32.466 ********** 2020-03-09T00:18:41.153 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-09T00:18:41.154 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-09T00:18:41.198 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:41 +0000 (0:00:00.704) 0:01:33.170 ********** 2020-03-09T00:18:41.202 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-09T00:18:41.245 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:41 +0000 (0:00:00.047) 0:01:33.218 ********** 2020-03-09T00:18:41.313 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-09T00:18:41.315 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-09T00:18:41.353 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:41 +0000 (0:00:00.107) 0:01:33.325 ********** 2020-03-09T00:18:41.602 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-09T00:18:41.640 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:41 +0000 (0:00:00.287) 0:01:33.613 ********** 2020-03-09T00:18:42.130 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-09T00:18:42.130 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-09T00:18:42.167 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:42 +0000 (0:00:00.527) 0:01:34.140 ********** 2020-03-09T00:18:42.188 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-09T00:18:42.225 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:42 +0000 (0:00:00.058) 0:01:34.198 ********** 2020-03-09T00:18:43.259 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-09T00:18:43.261 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-09T00:18:43.295 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:43 +0000 (0:00:01.069) 0:01:35.267 ********** 2020-03-09T00:18:43.324 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-09T00:18:43.371 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:43 +0000 (0:00:00.076) 0:01:35.343 ********** 2020-03-09T00:18:57.572 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-09T00:18:57.573 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.022700", "end": "2020-03-09 00:18:57.546354", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:18:41.523654", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-09T00:18:57.616 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:57 +0000 (0:00:14.245) 0:01:49.589 ********** 2020-03-09T00:18:58.104 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-09T00:18:58.105 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-09T00:18:58.142 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:58 +0000 (0:00:00.526) 0:01:50.115 ********** 2020-03-09T00:18:58.369 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-09T00:18:58.371 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.020498", "end": "2020-03-09 00:18:58.339572", "rc": 0, "start": "2020-03-09 00:18:58.319074", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:18:58.408 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:58 +0000 (0:00:00.265) 0:01:50.381 ********** 2020-03-09T00:18:58.606 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-09T00:18:58.607 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002205", "end": "2020-03-09 00:18:58.581965", "rc": 0, "start": "2020-03-09 00:18:58.579760", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:18:58.650 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:58 +0000 (0:00:00.241) 0:01:50.623 ********** 2020-03-09T00:18:58.656 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-09T00:18:58.657 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.779323", "end": "2020-03-09 00:18:58.601675", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:18:41.822352", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-09T00:18:58.700 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:58 +0000 (0:00:00.049) 0:01:50.672 ********** 2020-03-09T00:18:59.231 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-09T00:18:59.232 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-09T00:18:59.271 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:59 +0000 (0:00:00.571) 0:01:51.244 ********** 2020-03-09T00:18:59.502 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-09T00:18:59.503 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015653", "end": "2020-03-09 00:18:59.472420", "rc": 0, "start": "2020-03-09 00:18:59.456767", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:18:59.554 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:59 +0000 (0:00:00.282) 0:01:51.527 ********** 2020-03-09T00:18:59.762 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-09T00:18:59.762 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002156", "end": "2020-03-09 00:18:59.740679", "rc": 0, "start": "2020-03-09 00:18:59.738523", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:18:59.764 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-09T00:18:59.765 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.266514", "end": "2020-03-09 00:18:59.743817", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:18:43.477303", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-09T00:18:59.807 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:59 +0000 (0:00:00.252) 0:01:51.779 ********** 2020-03-09T00:18:59.842 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:59 +0000 (0:00:00.035) 0:01:51.815 ********** 2020-03-09T00:18:59.848 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.227380", "end": "2020-03-09 00:18:59.771326", "failed_when_result": false, "rc": 0, "start": "2020-03-09 00:18:43.543946", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-09T00:18:59.888 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:18:59 +0000 (0:00:00.046) 0:01:51.861 ********** 2020-03-09T00:19:00.316 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-09T00:19:00.318 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-09T00:19:00.360 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:00 +0000 (0:00:00.472) 0:01:52.333 ********** 2020-03-09T00:19:00.378 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-09T00:19:00.423 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:00 +0000 (0:00:00.062) 0:01:52.396 ********** 2020-03-09T00:19:00.590 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.023269", "end": "2020-03-09 00:19:00.564960", "rc": 0, "start": "2020-03-09 00:19:00.541691", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:00.631 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:00 +0000 (0:00:00.208) 0:01:52.604 ********** 2020-03-09T00:19:00.637 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013942", "end": "2020-03-09 00:19:00.610828", "rc": 0, "start": "2020-03-09 00:19:00.596886", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:00.680 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:00 +0000 (0:00:00.048) 0:01:52.652 ********** 2020-03-09T00:19:00.843 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-09T00:19:00.845 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002138", "end": "2020-03-09 00:19:00.814040", "rc": 0, "start": "2020-03-09 00:19:00.811902", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:00.914 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:00 +0000 (0:00:00.233) 0:01:52.886 ********** 2020-03-09T00:19:00.924 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002370", "end": "2020-03-09 00:19:00.855006", "rc": 0, "start": "2020-03-09 00:19:00.852636", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:00.965 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:00 +0000 (0:00:00.051) 0:01:52.938 ********** 2020-03-09T00:19:14.894 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-09T00:19:14.895 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.040993", "end": "2020-03-09 00:19:14.870635", "rc": 0, "start": "2020-03-09 00:18:58.829642", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:14.936 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:14 +0000 (0:00:13.970) 0:02:06.908 ********** 2020-03-09T00:19:15.134 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-09T00:19:15.136 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-09T00:19:15.173 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:15 +0000 (0:00:00.237) 0:02:07.146 ********** 2020-03-09T00:19:15.210 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-09T00:19:15.211 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-09T00:19:15.250 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:15 +0000 (0:00:00.076) 0:02:07.222 ********** 2020-03-09T00:19:15.294 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-09T00:19:15.296 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:15.332 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:15 +0000 (0:00:00.081) 0:02:07.304 ********** 2020-03-09T00:19:15.370 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-09T00:19:15.371 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-09T00:19:15.409 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:15 +0000 (0:00:00.077) 0:02:07.382 ********** 2020-03-09T00:19:15.447 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-09T00:19:15.449 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:15.484 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:15 +0000 (0:00:00.074) 0:02:07.457 ********** 2020-03-09T00:19:15.679 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-09T00:19:15.680 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-09T00:19:15.716 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:15 +0000 (0:00:00.231) 0:02:07.688 ********** 2020-03-09T00:19:15.924 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-09T00:19:15.926 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2020-03-09T00:19:15.976 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:15 +0000 (0:00:00.260) 0:02:07.949 ********** 2020-03-09T00:19:16.003 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-09T00:19:16.004 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:16.038 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:16 +0000 (0:00:00.061) 0:02:08.011 ********** 2020-03-09T00:19:16.064 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-09T00:19:16.066 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:16.106 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:16 +0000 (0:00:00.067) 0:02:08.079 ********** 2020-03-09T00:19:16.323 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-09T00:19:16.325 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-09T00:19:16.359 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:16 +0000 (0:00:00.252) 0:02:08.331 ********** 2020-03-09T00:19:16.732 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-09T00:19:16.732 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.716733", "end": "2020-03-09 00:19:16.711156", "rc": 0, "start": "2020-03-09 00:18:59.994423", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:16.770 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:16 +0000 (0:00:00.411) 0:02:08.743 ********** 2020-03-09T00:19:16.935 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-09T00:19:16.978 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:16 +0000 (0:00:00.207) 0:02:08.951 ********** 2020-03-09T00:19:16.994 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-09T00:19:16.995 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-09T00:19:17.032 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.054) 0:02:09.005 ********** 2020-03-09T00:19:17.038 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-09T00:19:17.038 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi104"}, "changed": false} 2020-03-09T00:19:17.078 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.045) 0:02:09.050 ********** 2020-03-09T00:19:17.084 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-09T00:19:17.085 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-09T00:19:17.124 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.046) 0:02:09.097 ********** 2020-03-09T00:19:17.162 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-09T00:19:17.164 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:17.203 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.078) 0:02:09.176 ********** 2020-03-09T00:19:17.209 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-09T00:19:17.210 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.014902", "end": "2020-03-09 00:19:17.157307", "rc": 0, "start": "2020-03-09 00:19:01.142405", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:17.212 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.081333", "end": "2020-03-09 00:19:17.171081", "rc": 0, "start": "2020-03-09 00:19:01.089748", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:17.252 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.049) 0:02:09.225 ********** 2020-03-09T00:19:17.291 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.038) 0:02:09.263 ********** 2020-03-09T00:19:17.298 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-09T00:19:17.299 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-09T00:19:17.340 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.049) 0:02:09.312 ********** 2020-03-09T00:19:17.379 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-09T00:19:17.381 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:17.418 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.077) 0:02:09.390 ********** 2020-03-09T00:19:17.466 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-09T00:19:17.467 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-09T00:19:17.508 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.090) 0:02:09.481 ********** 2020-03-09T00:19:17.515 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-09T00:19:17.556 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.047) 0:02:09.528 ********** 2020-03-09T00:19:17.561 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-09T00:19:17.562 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-09T00:19:17.602 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.046) 0:02:09.575 ********** 2020-03-09T00:19:17.608 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-09T00:19:17.648 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.046) 0:02:09.621 ********** 2020-03-09T00:19:17.653 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-09T00:19:17.654 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} TASK [testnode : Including distro specific variables.] ************************* ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:17.694 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.046) 0:02:09.667 ********** 2020-03-09T00:19:17.728 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.032) 0:02:09.700 ********** 2020-03-09T00:19:17.734 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:17.774 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.046) 0:02:09.747 ********** 2020-03-09T00:19:17.781 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-09T00:19:17.782 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-09T00:19:17.823 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.048) 0:02:09.796 ********** 2020-03-09T00:19:17.830 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-09T00:19:17.870 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.046) 0:02:09.843 ********** 2020-03-09T00:19:17.876 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-09T00:19:17.878 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:17.921 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.050) 0:02:09.894 ********** 2020-03-09T00:19:17.929 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-09T00:19:17.930 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2020-03-09T00:19:17.931 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-09T00:19:17.933 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-09T00:19:17.975 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:17 +0000 (0:00:00.053) 0:02:09.947 ********** 2020-03-09T00:19:18.012 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.036) 0:02:09.984 ********** 2020-03-09T00:19:18.017 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-09T00:19:18.018 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:18.059 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.046) 0:02:10.031 ********** 2020-03-09T00:19:18.085 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-09T00:19:18.088 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:18.128 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.069) 0:02:10.100 ********** 2020-03-09T00:19:18.134 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-09T00:19:18.134 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi104.front.sepia.ceph.com", "ansible_hostname": "smithi104", "ansible_nodename": "smithi104"}, "changed": true, "name": "smithi104"} 2020-03-09T00:19:18.136 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-09T00:19:18.177 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.049) 0:02:10.150 ********** 2020-03-09T00:19:18.213 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.035) 0:02:10.186 ********** 2020-03-09T00:19:18.218 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-09T00:19:18.258 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.045) 0:02:10.231 ********** 2020-03-09T00:19:18.361 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-09T00:19:18.362 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-09T00:19:18.405 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.146) 0:02:10.377 ********** 2020-03-09T00:19:18.413 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-09T00:19:18.414 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2020-03-09T00:19:18.481 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.076) 0:02:10.453 ********** 2020-03-09T00:19:18.488 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2020-03-09T00:19:18.528 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.047) 0:02:10.501 ********** 2020-03-09T00:19:18.535 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-09T00:19:18.536 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:18.577 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.048) 0:02:10.550 ********** 2020-03-09T00:19:18.584 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:18.630 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.053) 0:02:10.603 ********** 2020-03-09T00:19:18.646 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-09T00:19:18.648 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:18.686 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.055) 0:02:10.658 ********** 2020-03-09T00:19:18.693 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-09T00:19:18.694 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-09T00:19:18.696 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:18.735 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.049) 0:02:10.708 ********** 2020-03-09T00:19:18.775 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.039) 0:02:10.748 ********** 2020-03-09T00:19:18.900 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-09T00:19:18.901 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-09T00:19:18.939 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:18 +0000 (0:00:00.164) 0:02:10.912 ********** 2020-03-09T00:19:18.998 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-09T00:19:18.999 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-09T00:19:19.001 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-09T00:19:19.002 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-09T00:19:19.043 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.103) 0:02:11.015 ********** 2020-03-09T00:19:19.078 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.035) 0:02:11.050 ********** 2020-03-09T00:19:19.083 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-09T00:19:19.083 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi050"}, "changed": false} 2020-03-09T00:19:19.124 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.046) 0:02:11.097 ********** 2020-03-09T00:19:19.213 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-09T00:19:19.214 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-09T00:19:19.255 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.130) 0:02:11.227 ********** 2020-03-09T00:19:19.504 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-09T00:19:19.505 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-09T00:19:19.544 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.289) 0:02:11.517 ********** 2020-03-09T00:19:19.581 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-09T00:19:19.583 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi139"}, "changed": false} 2020-03-09T00:19:19.621 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.077) 0:02:11.594 ********** 2020-03-09T00:19:19.648 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-09T00:19:19.649 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-09T00:19:19.690 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.069) 0:02:11.663 ********** 2020-03-09T00:19:19.708 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-09T00:19:19.708 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi050", "ansible_hostname": "smithi050", "ansible_nodename": "smithi050"}, "changed": false, "name": "smithi050"} 2020-03-09T00:19:19.752 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.061) 0:02:11.725 ********** 2020-03-09T00:19:19.758 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-09T00:19:19.759 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi148"}, "changed": false} 2020-03-09T00:19:19.801 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.048) 0:02:11.774 ********** 2020-03-09T00:19:19.882 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-09T00:19:19.883 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-09T00:19:19.921 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:19 +0000 (0:00:00.119) 0:02:11.893 ********** 2020-03-09T00:19:20.189 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi139.front.sepia.ceph.com", "ansible_hostname": "smithi139", "ansible_nodename": "smithi139"}, "changed": true, "name": "smithi139"} 2020-03-09T00:19:20.229 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.307) 0:02:12.201 ********** 2020-03-09T00:19:20.267 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-09T00:19:20.268 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-09T00:19:20.308 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.079) 0:02:12.281 ********** 2020-03-09T00:19:20.368 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-09T00:19:20.369 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi148.front.sepia.ceph.com", "ansible_hostname": "smithi148", "ansible_nodename": "smithi148"}, "changed": true, "name": "smithi148"} 2020-03-09T00:19:20.412 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.104) 0:02:12.385 ********** 2020-03-09T00:19:20.418 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-09T00:19:20.419 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-09T00:19:20.459 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.046) 0:02:12.432 ********** 2020-03-09T00:19:20.664 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-09T00:19:20.665 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:19:20.703 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.244) 0:02:12.676 ********** 2020-03-09T00:19:20.709 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-09T00:19:20.710 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-09T00:19:20.753 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.050) 0:02:12.726 ********** 2020-03-09T00:19:20.802 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-09T00:19:20.803 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-09T00:19:20.843 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.089) 0:02:12.816 ********** 2020-03-09T00:19:20.870 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-09T00:19:20.871 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-09T00:19:20.909 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.065) 0:02:12.881 ********** 2020-03-09T00:19:20.914 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-09T00:19:20.915 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-09T00:19:20.956 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:20 +0000 (0:00:00.047) 0:02:12.929 ********** 2020-03-09T00:19:21.182 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-09T00:19:21.184 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-09T00:19:21.224 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:21 +0000 (0:00:00.267) 0:02:13.196 ********** 2020-03-09T00:19:21.229 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-09T00:19:21.230 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-09T00:19:21.269 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:21 +0000 (0:00:00.045) 0:02:13.242 ********** 2020-03-09T00:19:21.371 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-09T00:19:21.413 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:21 +0000 (0:00:00.143) 0:02:13.385 ********** 2020-03-09T00:19:21.451 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-09T00:19:21.451 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-09T00:19:21.490 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:21 +0000 (0:00:00.076) 0:02:13.462 ********** 2020-03-09T00:19:21.696 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-09T00:19:21.697 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-09T00:19:21.735 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:21 +0000 (0:00:00.245) 0:02:13.708 ********** 2020-03-09T00:19:21.898 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-09T00:19:21.900 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-09T00:19:21.935 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:21 +0000 (0:00:00.199) 0:02:13.907 ********** 2020-03-09T00:19:22.004 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-09T00:19:22.005 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-09T00:19:22.048 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.112) 0:02:14.020 ********** 2020-03-09T00:19:22.053 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-09T00:19:22.053 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-09T00:19:22.098 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.051) 0:02:14.071 ********** 2020-03-09T00:19:22.223 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-09T00:19:22.224 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-09T00:19:22.267 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.168) 0:02:14.240 ********** 2020-03-09T00:19:22.274 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-09T00:19:22.275 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:19:22.314 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.046) 0:02:14.286 ********** 2020-03-09T00:19:22.426 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-09T00:19:22.427 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-09T00:19:22.466 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.152) 0:02:14.439 ********** 2020-03-09T00:19:22.525 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-09T00:19:22.527 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-09T00:19:22.564 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.097) 0:02:14.537 ********** 2020-03-09T00:19:22.582 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-09T00:19:22.583 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-09T00:19:22.622 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.057) 0:02:14.594 ********** 2020-03-09T00:19:22.665 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-09T00:19:22.667 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:19:22.710 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.088) 0:02:14.682 ********** 2020-03-09T00:19:22.793 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-09T00:19:22.794 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-09T00:19:22.833 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.123) 0:02:14.806 ********** 2020-03-09T00:19:22.843 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-09T00:19:22.845 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:19:22.885 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.051) 0:02:14.858 ********** 2020-03-09T00:19:22.912 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-09T00:19:22.914 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-09T00:19:22.951 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:22 +0000 (0:00:00.065) 0:02:14.923 ********** 2020-03-09T00:19:23.084 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-09T00:19:23.122 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:23 +0000 (0:00:00.170) 0:02:15.094 ********** 2020-03-09T00:19:23.170 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-09T00:19:23.170 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-09T00:19:23.212 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:23 +0000 (0:00:00.090) 0:02:15.185 ********** 2020-03-09T00:19:23.345 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-09T00:19:23.346 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-09T00:19:23.347 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi148.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-09T00:19:23.393 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:23 +0000 (0:00:00.180) 0:02:15.365 ********** 2020-03-09T00:19:23.428 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:23 +0000 (0:00:00.035) 0:02:15.401 ********** 2020-03-09T00:19:23.713 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-09T00:19:23.715 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-09T00:19:23.757 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:23 +0000 (0:00:00.328) 0:02:15.729 ********** 2020-03-09T00:19:23.909 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-09T00:19:23.911 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-09T00:19:23.949 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:23 +0000 (0:00:00.191) 0:02:15.921 ********** 2020-03-09T00:19:23.955 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-09T00:19:23.956 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-09T00:19:23.996 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:23 +0000 (0:00:00.047) 0:02:15.969 ********** 2020-03-09T00:19:24.170 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-09T00:19:24.172 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.710908", "end": "2020-03-09 00:19:24.147857", "rc": 0, "start": "2020-03-09 00:19:22.436949", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:24.211 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:24 +0000 (0:00:00.215) 0:02:16.184 ********** 2020-03-09T00:19:24.238 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-09T00:19:24.239 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-09T00:19:24.278 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:24 +0000 (0:00:00.067) 0:02:16.251 ********** 2020-03-09T00:19:24.486 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-09T00:19:24.525 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:24 +0000 (0:00:00.246) 0:02:16.497 ********** 2020-03-09T00:19:24.710 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-09T00:19:24.711 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-09T00:19:24.754 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:24 +0000 (0:00:00.228) 0:02:16.726 ********** 2020-03-09T00:19:24.782 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-09T00:19:24.819 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:24 +0000 (0:00:00.064) 0:02:16.791 ********** 2020-03-09T00:19:25.408 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-09T00:19:25.409 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.374893", "end": "2020-03-09 00:19:25.377170", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-09 00:19:25.002277", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-09T00:19:25.446 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.627) 0:02:17.419 ********** 2020-03-09T00:19:25.475 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-09T00:19:25.477 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:25.514 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.067) 0:02:17.486 ********** 2020-03-09T00:19:25.552 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-09T00:19:25.552 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:19:25.563 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:19:25.602 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.088) 0:02:17.574 ********** 2020-03-09T00:19:25.631 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-09T00:19:25.632 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:25.668 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.066) 0:02:17.641 ********** 2020-03-09T00:19:25.705 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:19:25.708 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-09T00:19:25.749 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.080) 0:02:17.722 ********** 2020-03-09T00:19:25.787 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-09T00:19:25.788 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:25.826 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.076) 0:02:17.798 ********** 2020-03-09T00:19:25.866 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-09T00:19:25.867 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:25.907 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.081) 0:02:17.879 ********** 2020-03-09T00:19:25.913 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-09T00:19:25.914 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.730472", "end": "2020-03-09 00:19:25.862414", "rc": 0, "start": "2020-03-09 00:19:24.131942", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:25.951 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:25 +0000 (0:00:00.044) 0:02:17.924 ********** 2020-03-09T00:19:26.096 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.619567", "end": "2020-03-09 00:19:26.069080", "rc": 0, "start": "2020-03-09 00:19:24.449513", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:26.139 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.187) 0:02:18.111 ********** 2020-03-09T00:19:26.145 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-09T00:19:26.146 INFO:teuthology.task.ansible.out:fatal: [smithi104.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006702", "end": "2020-03-09 00:19:26.091859", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:19:26.085157", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-09T00:19:26.188 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.049) 0:02:18.161 ********** 2020-03-09T00:19:26.215 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-09T00:19:26.217 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:26.254 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.065) 0:02:18.226 ********** 2020-03-09T00:19:26.409 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-09T00:19:26.410 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.687398", "end": "2020-03-09 00:19:26.384998", "rc": 0, "start": "2020-03-09 00:19:24.697600", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:19:26.451 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.197) 0:02:18.424 ********** 2020-03-09T00:19:26.500 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-09T00:19:26.501 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-09T00:19:26.540 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.089) 0:02:18.513 ********** 2020-03-09T00:19:26.567 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-09T00:19:26.608 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.067) 0:02:18.581 ********** 2020-03-09T00:19:26.646 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi139.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-09T00:19:26.686 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.077) 0:02:18.658 ********** 2020-03-09T00:19:26.713 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-09T00:19:26.751 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.065) 0:02:18.724 ********** 2020-03-09T00:19:26.957 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-09T00:19:26.958 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-09T00:19:26.997 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:26 +0000 (0:00:00.246) 0:02:18.970 ********** 2020-03-09T00:19:27.024 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-09T00:19:27.061 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.063) 0:02:19.033 ********** 2020-03-09T00:19:27.130 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-09T00:19:27.132 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.300660", "end": "2020-03-09 00:19:27.099872", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-09 00:19:26.799212", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-09T00:19:27.171 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.109) 0:02:19.143 ********** 2020-03-09T00:19:27.199 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-09T00:19:27.200 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.237 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.066) 0:02:19.210 ********** 2020-03-09T00:19:27.274 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-09T00:19:27.276 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} TASK [testnode : Enable copr repos] ******************************************** 2020-03-09T00:19:27.278 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.317043", "end": "2020-03-09 00:19:27.251312", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-09 00:19:26.934269", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-09T00:19:27.279 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.318 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.079) 0:02:19.289 ********** 2020-03-09T00:19:27.354 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.037) 0:02:19.327 ********** 2020-03-09T00:19:27.360 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-09T00:19:27.362 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.405 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.046) 0:02:19.373 ********** TASK [testnode : Install GPG] ************************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.444 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.043) 0:02:19.416 ********** 2020-03-09T00:19:27.451 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:19:27.453 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-09T00:19:27.494 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.050) 0:02:19.466 ********** 2020-03-09T00:19:27.499 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-09T00:19:27.500 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.501 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.541 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.047) 0:02:19.514 ********** 2020-03-09T00:19:27.547 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-09T00:19:27.547 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.587 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.045) 0:02:19.560 ********** 2020-03-09T00:19:27.594 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-09T00:19:27.594 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.596 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** changed: [smithi148.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.325284", "end": "2020-03-09 00:19:27.569137", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-09 00:19:27.243853", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-09T00:19:27.635 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.047) 0:02:19.607 ********** 2020-03-09T00:19:27.667 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.032) 0:02:19.640 ********** 2020-03-09T00:19:27.672 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-09T00:19:27.672 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.674 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-09T00:19:27.712 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.044) 0:02:19.684 ********** 2020-03-09T00:19:27.747 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.034) 0:02:19.719 ********** 2020-03-09T00:19:27.753 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-09T00:19:27.754 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.792 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.045) 0:02:19.765 ********** 2020-03-09T00:19:27.797 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-09T00:19:27.797 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.844 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.051) 0:02:19.817 ********** 2020-03-09T00:19:27.850 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-09T00:19:27.850 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.852 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.891 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.047) 0:02:19.864 ********** 2020-03-09T00:19:27.896 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-09T00:19:27.898 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:27.936 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.044) 0:02:19.909 ********** 2020-03-09T00:19:27.941 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-09T00:19:27.943 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-09T00:19:27.945 INFO:teuthology.task.ansible.out:fatal: [smithi050.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006950", "end": "2020-03-09 00:19:27.912903", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:19:27.905953", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-09T00:19:27.981 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:27 +0000 (0:00:00.044) 0:02:19.954 ********** 2020-03-09T00:19:28.014 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.032) 0:02:19.987 ********** 2020-03-09T00:19:28.019 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-09T00:19:28.019 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:28.058 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.044) 0:02:20.031 ********** 2020-03-09T00:19:28.066 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:19:28.067 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-09T00:19:28.102 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.043) 0:02:20.075 ********** 2020-03-09T00:19:28.139 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-09T00:19:28.141 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:28.178 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.075) 0:02:20.151 ********** 2020-03-09T00:19:28.185 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-09T00:19:28.186 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006616", "end": "2020-03-09 00:19:28.124668", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:19:28.118052", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-09T00:19:28.224 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.046) 0:02:20.197 ********** 2020-03-09T00:19:28.229 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-09T00:19:28.231 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:28.278 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.053) 0:02:20.251 ********** 2020-03-09T00:19:28.284 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-09T00:19:28.285 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:28.326 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.047) 0:02:20.298 ********** 2020-03-09T00:19:28.491 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-09T00:19:28.492 INFO:teuthology.task.ansible.out:fatal: [smithi148.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006574", "end": "2020-03-09 00:19:28.469915", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:19:28.463341", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-09T00:19:28.539 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.213) 0:02:20.511 ********** 2020-03-09T00:19:28.556 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-09T00:19:28.556 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-09T00:19:28.595 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.056) 0:02:20.568 ********** 2020-03-09T00:19:28.602 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-09T00:19:28.604 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:28.641 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:28 +0000 (0:00:00.046) 0:02:20.614 ********** 2020-03-09T00:19:30.316 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-09T00:19:30.317 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-09T00:19:30.359 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:30 +0000 (0:00:01.717) 0:02:22.331 ********** 2020-03-09T00:19:30.640 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-09T00:19:30.680 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:30 +0000 (0:00:00.321) 0:02:22.653 ********** 2020-03-09T00:19:30.932 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-09T00:19:30.971 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:30 +0000 (0:00:00.291) 0:02:22.944 ********** 2020-03-09T00:19:31.799 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-09T00:19:31.800 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-09T00:19:31.837 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:31 +0000 (0:00:00.866) 0:02:23.810 ********** 2020-03-09T00:19:33.477 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-09T00:19:33.524 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:33 +0000 (0:00:01.686) 0:02:25.497 ********** 2020-03-09T00:19:33.801 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-09T00:19:33.840 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:33 +0000 (0:00:00.315) 0:02:25.812 ********** 2020-03-09T00:19:34.049 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-09T00:19:34.088 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:34 +0000 (0:00:00.248) 0:02:26.061 ********** 2020-03-09T00:19:44.877 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-09T00:19:44.879 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:12.805868", "end": "2020-03-09 00:19:44.850960", "rc": 0, "start": "2020-03-09 00:19:32.045092", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\""], "stdout": "CentOS-8 - AppStream 21 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 14 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 3.2 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 75 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 90 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:19:33 AM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-09T00:19:44.880 INFO:teuthology.task.ansible.out:===================\nUpgrading:\n systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M\n systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k\n systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k\n systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\nUpgrade 4 Packages\n\nTotal download size: 5.6 M\nDownloading Packages:\n(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.2 MB/s | 232 kB 00:00 \n(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 2.5 MB/s | 562 kB 00:00 \n(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 12 MB/s | 1.3 MB 00:00 \n(4/4): systemd-239-18.el8_1.4.x86_64.rpm 11 MB/s | 3.5 MB 00:00 \n--------------------------------------------------------------------------------\nTotal 11 MB/s | 5.6 MB 00:00 \nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 \n Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Run 2020-03-09T00:19:44.882 INFO:teuthology.task.ansible.out:ning scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Cleanup : systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 \n Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 \n Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 \n Verifying : systemd-239-18.el8_1.4.x86_64 1/8 \n Verifying : systemd-239-18.el8_1.2.x86_64 2/8 \n Verifying : systemd-libs-239-18.el8_1.4.x86_64 3/8 \n Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 \n Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 \n Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 \n Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 \n Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 \n\nUpgraded:\n systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 \n systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 \n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 21 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 14 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 3.2 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 75 kB/s | 3.3 kB 0 2020-03-09T00:19:44.883 INFO:teuthology.task.ansible.out:0:00 ", "Extra Packages for Enterprise Linux 90 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:19:33 AM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading:", " systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M", " systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k", " systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k", " systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "Upgrade 4 Packages", "", "Total download size: 5.6 M", "Downloading Packages:", "(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.2 MB/s | 232 kB 00:00 ", "(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 2.5 MB/s | 562 kB 00:00 ", "(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 12 MB/s | 1.3 MB 00:00 ", "(4/4): systemd-239-18.el8_1.4.x86_64.rpm 11 MB/s | 3 2020-03-09T00:19:44.885 INFO:teuthology.task.ansible.out:.5 MB 00:00 ", "--------------------------------------------------------------------------------", "Total 11 MB/s | 5.6 MB 00:00 ", "Running transaction check", "Transaction check succeeded.", "Running transaction test", "Transaction test succeeded.", "Running transaction", " Preparing : 1/1 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 ", " Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading : systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Cleanup : systemd-udev-239-18.el8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 ", " Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 ", " Verifying : systemd-239-18.el8_1.4.x86_64 2020-03-09T00:19:44.887 INFO:teuthology.task.ansible.out: 1/8 ", " Verifying : systemd-239-18.el8_1.2.x86_64 2/8 ", " Verifying : systemd-libs-239-18.el8_1.4.x86_64 3/8 ", " Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 ", " Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 ", " Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 ", " Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 ", " Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 ", "", "Upgraded:", " systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 ", " systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 ", "", "Complete!"]} 2020-03-09T00:19:44.916 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:44 +0000 (0:00:10.828) 0:02:36.889 ********** 2020-03-09T00:19:46.522 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:12.183941", "end": "2020-03-09 00:19:46.492023", "rc": 0, "start": "2020-03-09 00:19:34.308082", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\""], "stdout": "CentOS-8 - AppStream 14 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 14 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 4.6 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 59 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 139 kB/s | 5.3 kB 00:00 \nlab-extras 1.1 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:03 ago on Mon 09 Mar 2020 12:19:35 AM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-09T00:19:46.524 INFO:teuthology.task.ansible.out:===================\nUpgrading:\n systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M\n systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k\n systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k\n systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\nUpgrade 4 Packages\n\nTotal download size: 5.6 M\nDownloading Packages:\n(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.2 MB/s | 232 kB 00:00 \n(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 2.5 MB/s | 562 kB 00:00 \n(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 12 MB/s | 1.3 MB 00:00 \n(4/4): systemd-239-18.el8_1.4.x86_64.rpm 9.9 MB/s | 3.5 MB 00:00 \n--------------------------------------------------------------------------------\nTotal 11 MB/s | 5.6 MB 00:00 \nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 \n Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Run 2020-03-09T00:19:46.526 INFO:teuthology.task.ansible.out:ning scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Cleanup : systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 \n Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 \n Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 \n Verifying : systemd-239-18.el8_1.4.x86_64 1/8 \n Verifying : systemd-239-18.el8_1.2.x86_64 2/8 \n Verifying : systemd-libs-239-18.el8_1.4.x86_64 3/8 \n Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 \n Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 \n Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 \n Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 \n Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 \n\nUpgraded:\n systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 \n systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 \n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 14 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 14 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 4.6 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 59 kB/s | 3.3 kB 0 2020-03-09T00:19:46.527 INFO:teuthology.task.ansible.out:0:00 ", "Extra Packages for Enterprise Linux 139 kB/s | 5.3 kB 00:00 ", "lab-extras 1.1 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:03 ago on Mon 09 Mar 2020 12:19:35 AM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading:", " systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M", " systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k", " systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k", " systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "Upgrade 4 Packages", "", "Total download size: 5.6 M", "Downloading Packages:", "(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.2 MB/s | 232 kB 00:00 ", "(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 2.5 MB/s | 562 kB 00:00 ", "(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 12 MB/s | 1.3 MB 00:00 ", "(4/4): systemd-239-18.el8_1.4.x86_64.rpm 9.9 MB/s | 3 2020-03-09T00:19:46.529 INFO:teuthology.task.ansible.out:.5 MB 00:00 ", "--------------------------------------------------------------------------------", "Total 11 MB/s | 5.6 MB 00:00 ", "Running transaction check", "Transaction check succeeded.", "Running transaction test", "Transaction test succeeded.", "Running transaction", " Preparing : 1/1 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 ", " Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading : systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Cleanup : systemd-udev-239-18.el8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 ", " Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 ", " Verifying : systemd-239-18.el8_1.4.x86_64 2020-03-09T00:19:46.531 INFO:teuthology.task.ansible.out: 1/8 ", " Verifying : systemd-239-18.el8_1.2.x86_64 2/8 ", " Verifying : systemd-libs-239-18.el8_1.4.x86_64 3/8 ", " Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 ", " Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 ", " Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 ", " Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 ", " Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 ", "", "Upgraded:", " systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 ", " systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 ", "", "Complete!"]} 2020-03-09T00:19:46.564 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:46 +0000 (0:00:01.647) 0:02:38.537 ********** 2020-03-09T00:19:46.931 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:13.155357", "end": "2020-03-09 00:19:46.898917", "rc": 0, "start": "2020-03-09 00:19:33.743560", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\""], "stdout": "CentOS-8 - AppStream 36 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 13 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 3.6 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 78 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 166 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:19:35 AM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-09T00:19:46.933 INFO:teuthology.task.ansible.out:===================\nUpgrading:\n systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M\n systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k\n systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k\n systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\nUpgrade 4 Packages\n\nTotal download size: 5.6 M\nDownloading Packages:\n(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.2 MB/s | 232 kB 00:00 \n(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 2.5 MB/s | 562 kB 00:00 \n(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 12 MB/s | 1.3 MB 00:00 \n(4/4): systemd-239-18.el8_1.4.x86_64.rpm 11 MB/s | 3.5 MB 00:00 \n--------------------------------------------------------------------------------\nTotal 10 MB/s | 5.6 MB 00:00 \nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 \n Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Run 2020-03-09T00:19:46.935 INFO:teuthology.task.ansible.out:ning scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Cleanup : systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 \n Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 \n Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 \n Verifying : systemd-239-18.el8_1.4.x86_64 1/8 \n Verifying : systemd-239-18.el8_1.2.x86_64 2/8 \n Verifying : systemd-libs-239-18.el8_1.4.x86_64 3/8 \n Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 \n Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 \n Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 \n Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 \n Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 \n\nUpgraded:\n systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 \n systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 \n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 36 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 13 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 3.6 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 78 kB/s | 3.3 kB 0 2020-03-09T00:19:46.936 INFO:teuthology.task.ansible.out:0:00 ", "Extra Packages for Enterprise Linux 166 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:19:35 AM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading:", " systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M", " systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k", " systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k", " systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "Upgrade 4 Packages", "", "Total download size: 5.6 M", "Downloading Packages:", "(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.2 MB/s | 232 kB 00:00 ", "(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 2.5 MB/s | 562 kB 00:00 ", "(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 12 MB/s | 1.3 MB 00:00 ", "(4/4): systemd-239-18.el8_1.4.x86_64.rpm 11 MB/s | 3 2020-03-09T00:19:46.938 INFO:teuthology.task.ansible.out:.5 MB 00:00 ", "--------------------------------------------------------------------------------", "Total 10 MB/s | 5.6 MB 00:00 ", "Running transaction check", "Transaction check succeeded.", "Running transaction test", "Transaction test succeeded.", "Running transaction", " Preparing : 1/1 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 ", " Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading : systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Cleanup : systemd-udev-239-18.e 2020-03-09T00:19:46.938 INFO:teuthology.task.ansible.out:l8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 ", " Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 ", " Verifying : systemd-239-18.el8_1.4.x86_64 2020-03-09T00:19:46.940 INFO:teuthology.task.ansible.out: 1/8 ", " Verifying : systemd-239-18.el8_1.2.x86_64 2/8 ", " Verifying : systemd-libs-239-18.el8_1.4.x86_64 3/8 ", " Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 ", " Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 ", " Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 ", " Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 ", " Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 ", "", "Upgraded:", " systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 ", " systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 ", "", "Complete!"]} 2020-03-09T00:19:46.977 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:46 +0000 (0:00:00.413) 0:02:38.950 ********** 2020-03-09T00:19:47.685 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:13.632470", "end": "2020-03-09 00:19:47.662433", "rc": 0, "start": "2020-03-09 00:19:34.029963", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\""], "stdout": "CentOS-8 - AppStream 29 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 33 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 3.2 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 70 kB/s | 3.3 kB 00:00 \nExt 2020-03-09T00:19:47.687 INFO:teuthology.task.ansible.out:ra Packages for Enterprise Linux 139 kB/s | 5.3 kB 00:00 \nlab-extras 1.3 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:19:35 AM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nUpgrading:\n systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M\n systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k\n systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k\n systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\nUpgrade 4 Packages\n\nTotal download size: 5.6 M\nDownloading Packages:\n(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.0 MB/s | 232 kB 00:00 \n(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 1.6 MB/s | 562 kB 00:00 \n(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 2.9 MB/s | 1.3 MB 00:00 \n(4/4): systemd-239-18.el8_1.4.x86_64.rpm 3.2 MB/s | 3.5 MB 00:01 \n---------------------------------------------------------------- 2020-03-09T00:19:47.688 INFO:teuthology.task.ansible.out:----------------\nTotal 4.4 MB/s | 5.6 MB 00:01 \nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 \n Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 \n Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 \n Upgrading : systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 \n Cleanup : systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 \n Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 \n Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 \n Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 \n Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 \n Verifying : systemd-239-18.el8_1.4.x86_64 1/8 \n Verifying : systemd-239-18.el8_1.2.x86_64 2/8 \n Verifying : systemd-libs-239-18.el8_ 2020-03-09T00:19:47.690 INFO:teuthology.task.ansible.out:1.4.x86_64 3/8 \n Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 \n Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 \n Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 \n Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 \n Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 \n\nUpgraded:\n systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 \n systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 \n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 29 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 33 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 3.2 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 70 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 139 kB/s | 5.3 kB 00:00 ", "lab-extras 1.3 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:02 ago on Mon 09 Mar 2020 12:19:35 AM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "============================================================ 2020-03-09T00:19:47.691 INFO:teuthology.task.ansible.out:====================", "Upgrading:", " systemd x86_64 239-18.el8_1.4 BaseOS 3.5 M", " systemd-libs x86_64 239-18.el8_1.4 BaseOS 562 k", " systemd-pam x86_64 239-18.el8_1.4 BaseOS 232 k", " systemd-udev x86_64 239-18.el8_1.4 BaseOS 1.3 M", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "Upgrade 4 Packages", "", "Total download size: 5.6 M", "Downloading Packages:", "(1/4): systemd-pam-239-18.el8_1.4.x86_64.rpm 1.0 MB/s | 232 kB 00:00 ", "(2/4): systemd-libs-239-18.el8_1.4.x86_64.rpm 1.6 MB/s | 562 kB 00:00 ", "(3/4): systemd-udev-239-18.el8_1.4.x86_64.rpm 2.9 MB/s | 1.3 MB 00:00 ", "(4/4): systemd-239-18.el8_1.4.x86_64.rpm 3.2 MB/s | 3.5 MB 00:01 ", "--------------------------------------------------------------------------------", "Total 4.4 MB/s | 5.6 MB 00:01 ", "Running transaction check", "Transaction check succeeded.", "Running transaction test", "Transaction test succeeded.", "Running transaction", " Preparing : 1/1 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/1 ", " Upgrading : systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Running scriptlet: systemd-libs-239-18.el8_1.4.x86_64 1/8 ", " Upgrading : systemd-pam-239-18.el8_1.4.x86_64 2/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading : systemd-239-18.el8_1.4.x86_64 3/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 3/8 ", " Upgrading 2020-03-09T00:19:47.693 INFO:teuthology.task.ansible.out: : systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 4/8 ", " Cleanup : systemd-udev-239-18.el8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 5/8 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-239-18.el8_1.2.x86_64 6/8 ", " Cleanup : systemd-libs-239-18.el8_1.2.x86_64 7/8 ", " Cleanup : systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-pam-239-18.el8_1.2.x86_64 8/8 ", " Running scriptlet: systemd-239-18.el8_1.4.x86_64 8/8 ", " Running scriptlet: systemd-udev-239-18.el8_1.4.x86_64 8/8 ", " Verifying : systemd-239-18.el8_1.4.x86_64 1/8 ", " Verifying : systemd-239-18.el8_1.2.x86_64 2/8 ", " Verifying : systemd-libs-239-18.el8_1.4.x86_64 3/8 ", " Verifying : systemd-libs-239-18.el8_1.2.x86_64 4/8 ", " Verifying : systemd-pam-239-18.el8_1.4.x86_64 5/8 ", " Verifying : systemd-pam-239-18.el8_1.2.x86_64 6/8 ", " Verifying : systemd-udev-239-18.el8_1.4.x86_64 7/8 ", " Verifying : systemd-udev-239-18.el8_1.2.x86_64 8/8 ", "", "Upgraded:", " systemd-239-18.el8_1.4.x86_64 systemd-libs-239-18.el8_1.4.x86_64 ", " systemd-pam-239-18.el8_1.4.x86_64 systemd-udev-239-18.el8_1.4.x86_64 ", "", "Complete!"]} 2020-03-09T00:19:47.729 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:47 +0000 (0:00:00.751) 0:02:39.701 ********** 2020-03-09T00:19:47.730 INFO:teuthology.task.ansible.out: 2020-03-09T00:19:57.142 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-09T00:19:57.143 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: NetworkManager-team-1:1.20.0-5.el8_1.x86_64", "Installed: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: NetworkManager-1:1.20.0-5.el8_1.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64", "Installed: hdparm-9.54-2.el8.x86_64", "Removed: NetworkManager-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-libnm-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-team-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-tui-1:1.20.0-3.el8.x86_64"]} 2020-03-09T00:19:57.180 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:57 +0000 (0:00:09.451) 0:02:49.153 ********** 2020-03-09T00:19:57.218 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-09T00:19:57.220 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:57.259 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:57 +0000 (0:00:00.079) 0:02:49.232 ********** 2020-03-09T00:19:57.850 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-09T00:19:57.851 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: NetworkManager-team-1:1.20.0-5.el8_1.x86_64", "Installed: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: NetworkManager-1:1.20.0-5.el8_1.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64", "Installed: hdparm-9.54-2.el8.x86_64", "Removed: NetworkManager-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-libnm-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-team-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-tui-1:1.20.0-3.el8.x86_64"]} 2020-03-09T00:19:57.892 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:57 +0000 (0:00:00.632) 0:02:49.864 ********** 2020-03-09T00:19:57.929 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-09T00:19:57.930 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:57.967 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:57 +0000 (0:00:00.075) 0:02:49.940 ********** 2020-03-09T00:19:58.175 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-09T00:19:58.176 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-09T00:19:58.216 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:58 +0000 (0:00:00.248) 0:02:50.188 ********** 2020-03-09T00:19:58.245 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:19:58.247 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:58.284 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:58 +0000 (0:00:00.068) 0:02:50.257 ********** 2020-03-09T00:19:58.522 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-09T00:19:58.523 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: NetworkManager-team-1:1.20.0-5.el8_1.x86_64", "Installed: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: NetworkManager-1:1.20.0-5.el8_1.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64", "Installed: hdparm-9.54-2.el8.x86_64", "Removed: NetworkManager-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-libnm-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-team-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-tui-1:1.20.0-3.el8.x86_64"]} 2020-03-09T00:19:58.564 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:58 +0000 (0:00:00.279) 0:02:50.536 ********** 2020-03-09T00:19:58.594 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-09T00:19:58.596 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service sysinit.target basic.target system.slice dbus.socket polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ipset.service shutdown.target ip6tables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-09T00:19:58.598 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127443", "LimitNPROCSoft": "127443", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127443", 2020-03-09T00:19:58.599 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127443", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-09T00:19:58.601 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} TASK [testnode : Install epel packages] **************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:58.639 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:58 +0000 (0:00:00.075) 0:02:50.612 ********** 2020-03-09T00:19:58.673 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:58 +0000 (0:00:00.034) 0:02:50.646 ********** 2020-03-09T00:19:58.939 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-09T00:19:58.941 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-09T00:19:58.978 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:58 +0000 (0:00:00.304) 0:02:50.950 ********** 2020-03-09T00:19:58.984 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-09T00:19:58.985 INFO:teuthology.task.ansible.out:fatal: [smithi104.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-09T00:19:59.027 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.048) 0:02:50.999 ********** 2020-03-09T00:19:59.031 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:19:59.032 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:59.071 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.044) 0:02:51.044 ********** 2020-03-09T00:19:59.390 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-09T00:19:59.395 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service sysinit.target dbus.socket system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service shutdown.target ipset.service ip6tables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-09T00:19:59.397 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127437", "LimitNPROCSoft": "127437", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127437", 2020-03-09T00:19:59.398 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127437", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-09T00:19:59.400 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-09T00:19:59.437 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.365) 0:02:51.410 ********** 2020-03-09T00:19:59.519 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-09T00:19:59.520 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: NetworkManager-team-1:1.20.0-5.el8_1.x86_64", "Installed: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: NetworkManager-1:1.20.0-5.el8_1.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64", "Installed: hdparm-9.54-2.el8.x86_64", "Removed: NetworkManager-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-libnm-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-team-1:1.20.0-3.el8.x86_64", "Removed: NetworkManager-tui-1:1.20.0-3.el8.x86_64"]} 2020-03-09T00:19:59.561 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.124) 0:02:51.534 ********** 2020-03-09T00:19:59.603 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-09T00:19:59.605 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:59.645 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.083) 0:02:51.617 ********** 2020-03-09T00:19:59.651 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-09T00:19:59.652 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-09T00:19:59.705 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.060) 0:02:51.678 ********** 2020-03-09T00:19:59.707 INFO:teuthology.task.ansible.out: 2020-03-09T00:19:59.735 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-09T00:19:59.736 INFO:teuthology.task.ansible.out:fatal: [smithi148.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-09T00:19:59.778 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.073) 0:02:51.751 ********** 2020-03-09T00:19:59.784 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:19:59.785 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:19:59.786 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-09T00:19:59.829 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.050) 0:02:51.801 ********** 2020-03-09T00:19:59.877 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:19:59 +0000 (0:00:00.047) 0:02:51.849 ********** 2020-03-09T00:20:00.040 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-09T00:20:00.042 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.205 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.328) 0:02:52.177 ********** 2020-03-09T00:20:00.212 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-09T00:20:00.212 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-09T00:20:00.254 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.049) 0:02:52.227 ********** 2020-03-09T00:20:00.260 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-09T00:20:00.304 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.049) 0:02:52.276 ********** 2020-03-09T00:20:00.310 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-09T00:20:00.312 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.312 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-09T00:20:00.315 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target sysinit.target system.slice polkit.service dbus.service dbus.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service ip6tables.service iptables.service ebtables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-09T00:20:00.317 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127467", "LimitNPROCSoft": "127467", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127467", 2020-03-09T00:20:00.318 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127467", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice dbus.socket sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-09T00:20:00.320 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-09T00:20:00.355 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.050) 0:02:52.327 ********** 2020-03-09T00:20:00.391 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.036) 0:02:52.364 ********** 2020-03-09T00:20:00.398 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-09T00:20:00.400 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.441 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.049) 0:02:52.414 ********** 2020-03-09T00:20:00.447 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-09T00:20:00.486 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.045) 0:02:52.459 ********** 2020-03-09T00:20:00.492 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-09T00:20:00.493 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.532 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.045) 0:02:52.505 ********** 2020-03-09T00:20:00.539 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-09T00:20:00.541 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.579 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.046) 0:02:52.551 ********** 2020-03-09T00:20:00.583 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-09T00:20:00.584 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.586 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.626 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.046) 0:02:52.598 ********** 2020-03-09T00:20:00.632 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-09T00:20:00.633 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-09T00:20:00.635 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.675 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.049) 0:02:52.647 ********** 2020-03-09T00:20:00.711 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.036) 0:02:52.684 ********** 2020-03-09T00:20:00.717 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-09T00:20:00.719 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi050.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-09T00:20:00.719 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:20:00.719 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.761 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.050) 0:02:52.734 ********** 2020-03-09T00:20:00.796 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.034) 0:02:52.769 ********** 2020-03-09T00:20:00.830 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.034) 0:02:52.803 ********** 2020-03-09T00:20:00.846 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi148.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} TASK [testnode : Add local apt repos.] ***************************************** 2020-03-09T00:20:00.879 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.048) 0:02:52.851 ********** 2020-03-09T00:20:00.913 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.034) 0:02:52.886 ********** 2020-03-09T00:20:00.919 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-09T00:20:00.920 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:00.958 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.044) 0:02:52.930 ********** 2020-03-09T00:20:00.963 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-09T00:20:00.967 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.005 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:00 +0000 (0:00:00.047) 0:02:52.978 ********** 2020-03-09T00:20:01.011 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-09T00:20:01.013 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.050 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.045) 0:02:53.023 ********** 2020-03-09T00:20:01.057 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-09T00:20:01.058 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-09T00:20:01.058 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-09T00:20:01.098 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.047) 0:02:53.070 ********** 2020-03-09T00:20:01.132 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.034) 0:02:53.105 ********** 2020-03-09T00:20:01.138 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-09T00:20:01.139 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for custom repos] *************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.178 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.045) 0:02:53.151 ********** 2020-03-09T00:20:01.213 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.034) 0:02:53.185 ********** 2020-03-09T00:20:01.227 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target dbus.socket polkit.service dbus.service basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service ip6tables.service shutdown.target iptables.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": " 2020-03-09T00:20:01.228 INFO:teuthology.task.ansible.out:/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Li 2020-03-09T00:20:01.230 INFO:teuthology.task.ansible.out:mitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWh 2020-03-09T00:20:01.233 INFO:teuthology.task.ansible.out:enUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Remove custom repos] ****************************************** 2020-03-09T00:20:01.264 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.051) 0:02:53.237 ********** 2020-03-09T00:20:01.299 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.034) 0:02:53.271 ********** 2020-03-09T00:20:01.335 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.036) 0:02:53.308 ********** 2020-03-09T00:20:01.342 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:20:01.344 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.384 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.049) 0:02:53.357 ********** 2020-03-09T00:20:01.423 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.038) 0:02:53.396 ********** 2020-03-09T00:20:01.566 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add sources list] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 09 March 2020 00:20:01 +0000 (0:00:00.047) 0:02:53.443 ********** Monday 09 March 2020 00:20:01 +0000 (0:00:00.034) 0:02:53.478 ********** TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install apt keys] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} Monday 09 March 2020 00:20:01 +0000 (0:00:00.056) 0:02:53.534 ********** 2020-03-09T00:20:01.595 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.033) 0:02:53.568 ********** 2020-03-09T00:20:01.603 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-09T00:20:01.603 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.605 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.649 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.053) 0:02:53.622 ********** 2020-03-09T00:20:01.685 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.035) 0:02:53.658 ********** 2020-03-09T00:20:01.690 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-09T00:20:01.691 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-09T00:20:01.693 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add local apt repos.] ***************************************** 2020-03-09T00:20:01.731 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.046) 0:02:53.704 ********** 2020-03-09T00:20:01.768 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.036) 0:02:53.741 ********** 2020-03-09T00:20:01.802 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.033) 0:02:53.775 ********** 2020-03-09T00:20:01.808 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:20:01.810 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.850 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.047) 0:02:53.823 ********** 2020-03-09T00:20:01.888 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.037) 0:02:53.860 ********** 2020-03-09T00:20:01.924 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.036) 0:02:53.896 ********** 2020-03-09T00:20:01.930 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-09T00:20:01.932 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:01.972 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:01 +0000 (0:00:00.047) 0:02:53.944 ********** 2020-03-09T00:20:02.006 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:53.979 ********** 2020-03-09T00:20:02.013 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-09T00:20:02.015 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.054 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.047) 0:02:54.026 ********** 2020-03-09T00:20:02.092 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.037) 0:02:54.064 ********** 2020-03-09T00:20:02.096 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-09T00:20:02.098 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.098 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-09T00:20:02.098 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-09T00:20:02.138 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.046) 0:02:54.110 ********** 2020-03-09T00:20:02.175 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.037) 0:02:54.148 ********** 2020-03-09T00:20:02.210 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:54.182 ********** 2020-03-09T00:20:02.215 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-09T00:20:02.217 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.257 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.047) 0:02:54.230 ********** 2020-03-09T00:20:02.292 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.033) 0:02:54.264 ********** 2020-03-09T00:20:02.325 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.033) 0:02:54.298 ********** 2020-03-09T00:20:02.331 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-09T00:20:02.333 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install packages] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.377 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.051) 0:02:54.349 ********** 2020-03-09T00:20:02.411 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:54.384 ********** 2020-03-09T00:20:02.446 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:54.419 ********** 2020-03-09T00:20:02.453 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-09T00:20:02.455 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.492 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.045) 0:02:54.464 ********** 2020-03-09T00:20:02.526 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:54.499 ********** 2020-03-09T00:20:02.563 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.037) 0:02:54.536 ********** 2020-03-09T00:20:02.569 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-09T00:20:02.570 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-09T00:20:02.571 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.572 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.610 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.047) 0:02:54.583 ********** 2020-03-09T00:20:02.645 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:54.617 ********** 2020-03-09T00:20:02.679 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:54.652 ********** 2020-03-09T00:20:02.685 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-09T00:20:02.687 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Stop apache2] ************************************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.726 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.046) 0:02:54.699 ********** 2020-03-09T00:20:02.763 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.036) 0:02:54.736 ********** 2020-03-09T00:20:02.798 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.034) 0:02:54.770 ********** 2020-03-09T00:20:02.803 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-09T00:20:02.804 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.806 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.843 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.045) 0:02:54.816 ********** 2020-03-09T00:20:02.879 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.035) 0:02:54.852 ********** 2020-03-09T00:20:02.917 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.038) 0:02:54.890 ********** 2020-03-09T00:20:02.923 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-09T00:20:02.924 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.925 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:02.967 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.050) 0:02:54.940 ********** 2020-03-09T00:20:03.005 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:02 +0000 (0:00:00.037) 0:02:54.978 ********** 2020-03-09T00:20:03.041 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.035) 0:02:55.013 ********** 2020-03-09T00:20:03.046 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-09T00:20:03.048 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.087 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.046) 0:02:55.060 ********** 2020-03-09T00:20:03.123 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.036) 0:02:55.096 ********** 2020-03-09T00:20:03.161 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.037) 0:02:55.134 ********** 2020-03-09T00:20:03.166 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-09T00:20:03.167 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.212 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.050) 0:02:55.184 ********** 2020-03-09T00:20:03.245 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.032) 0:02:55.217 ********** 2020-03-09T00:20:03.279 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.033) 0:02:55.251 ********** 2020-03-09T00:20:03.283 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-09T00:20:03.284 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.285 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.287 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.325 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.046) 0:02:55.297 ********** 2020-03-09T00:20:03.365 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.038) 0:02:55.335 ********** 2020-03-09T00:20:03.404 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.041) 0:02:55.376 ********** 2020-03-09T00:20:03.409 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-09T00:20:03.412 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.449 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.045) 0:02:55.422 ********** 2020-03-09T00:20:03.486 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.036) 0:02:55.458 ********** 2020-03-09T00:20:03.522 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.036) 0:02:55.495 ********** 2020-03-09T00:20:03.528 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-09T00:20:03.529 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.530 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-09T00:20:03.531 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.574 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.052) 0:02:55.547 ********** 2020-03-09T00:20:03.613 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.038) 0:02:55.585 ********** 2020-03-09T00:20:03.649 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.035) 0:02:55.621 ********** 2020-03-09T00:20:03.666 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.696 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.047) 0:02:55.669 ********** 2020-03-09T00:20:03.740 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.043) 0:02:55.712 ********** 2020-03-09T00:20:03.778 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.038) 0:02:55.750 ********** 2020-03-09T00:20:03.784 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-09T00:20:03.785 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-09T00:20:03.785 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.786 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.825 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.047) 0:02:55.798 ********** 2020-03-09T00:20:03.863 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.037) 0:02:55.836 ********** 2020-03-09T00:20:03.899 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.035) 0:02:55.871 ********** 2020-03-09T00:20:03.905 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-09T00:20:03.905 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.907 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install packages] ********************************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:03.944 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.045) 0:02:55.917 ********** 2020-03-09T00:20:03.983 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:03 +0000 (0:00:00.037) 0:02:55.954 ********** 2020-03-09T00:20:04.019 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.036) 0:02:55.991 ********** 2020-03-09T00:20:04.024 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-09T00:20:04.025 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.026 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.068 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.049) 0:02:56.041 ********** 2020-03-09T00:20:04.103 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.035) 0:02:56.076 ********** 2020-03-09T00:20:04.139 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.035) 0:02:56.111 ********** 2020-03-09T00:20:04.144 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-09T00:20:04.146 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.146 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-09T00:20:04.146 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.187 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.048) 0:02:56.159 ********** 2020-03-09T00:20:04.222 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.035) 0:02:56.195 ********** 2020-03-09T00:20:04.258 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.036) 0:02:56.231 ********** 2020-03-09T00:20:04.264 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-09T00:20:04.265 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.267 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.305 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.046) 0:02:56.278 ********** 2020-03-09T00:20:04.339 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.034) 0:02:56.312 ********** 2020-03-09T00:20:04.345 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-09T00:20:04.346 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.347 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.389 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.049) 0:02:56.361 ********** 2020-03-09T00:20:04.423 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.034) 0:02:56.396 ********** 2020-03-09T00:20:04.430 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-09T00:20:04.430 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.432 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.470 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.047) 0:02:56.443 ********** 2020-03-09T00:20:04.504 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.033) 0:02:56.477 ********** 2020-03-09T00:20:04.510 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:04.511 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-09T00:20:04.512 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.549 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.044) 0:02:56.522 ********** 2020-03-09T00:20:04.582 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.032) 0:02:56.555 ********** 2020-03-09T00:20:04.618 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.036) 0:02:56.591 ********** 2020-03-09T00:20:04.623 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:20:04.625 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.664 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.045) 0:02:56.636 ********** 2020-03-09T00:20:04.696 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.032) 0:02:56.669 ********** 2020-03-09T00:20:04.703 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-09T00:20:04.703 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.741 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.044) 0:02:56.714 ********** 2020-03-09T00:20:04.746 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-09T00:20:04.747 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.786 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.045) 0:02:56.759 ********** 2020-03-09T00:20:04.792 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-09T00:20:04.793 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.832 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.045) 0:02:56.805 ********** 2020-03-09T00:20:04.837 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:04.838 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:04.839 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.878 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.045) 0:02:56.850 ********** 2020-03-09T00:20:04.910 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.032) 0:02:56.883 ********** 2020-03-09T00:20:04.915 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-09T00:20:04.916 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:04.918 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-09T00:20:04.955 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.044) 0:02:56.928 ********** 2020-03-09T00:20:04.989 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:04 +0000 (0:00:00.033) 0:02:56.961 ********** 2020-03-09T00:20:04.993 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-09T00:20:04.995 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.033 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.044) 0:02:57.006 ********** 2020-03-09T00:20:05.039 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-09T00:20:05.041 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.078 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.044) 0:02:57.051 ********** 2020-03-09T00:20:05.083 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-09T00:20:05.083 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.085 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-09T00:20:05.129 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.051) 0:02:57.102 ********** 2020-03-09T00:20:05.167 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.037) 0:02:57.140 ********** 2020-03-09T00:20:05.174 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-09T00:20:05.175 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.177 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-09T00:20:05.178 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.217 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.049) 0:02:57.189 ********** 2020-03-09T00:20:05.257 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.040) 0:02:57.230 ********** 2020-03-09T00:20:05.259 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:05.263 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:05.265 INFO:teuthology.task.ansible.out:TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.266 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:05.268 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.308 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.050) 0:02:57.280 ********** 2020-03-09T00:20:05.309 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:05.342 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.034) 0:02:57.315 ********** 2020-03-09T00:20:05.381 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.038) 0:02:57.353 ********** 2020-03-09T00:20:05.387 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-09T00:20:05.389 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.433 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.052) 0:02:57.406 ********** 2020-03-09T00:20:05.473 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.039) 0:02:57.446 ********** 2020-03-09T00:20:05.480 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-09T00:20:05.480 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.481 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:05.481 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.521 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.047) 0:02:57.494 ********** 2020-03-09T00:20:05.570 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.049) 0:02:57.543 ********** 2020-03-09T00:20:05.576 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-09T00:20:05.577 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.579 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:05.580 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-09T00:20:05.582 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.621 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.050) 0:02:57.594 ********** 2020-03-09T00:20:05.664 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.043) 0:02:57.637 ********** 2020-03-09T00:20:05.700 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.036) 0:02:57.673 ********** 2020-03-09T00:20:05.702 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:05.704 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-09T00:20:05.706 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.707 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.745 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.045) 0:02:57.718 ********** 2020-03-09T00:20:05.780 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.034) 0:02:57.753 ********** 2020-03-09T00:20:05.817 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.036) 0:02:57.789 ********** 2020-03-09T00:20:05.824 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-09T00:20:05.825 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.827 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.828 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.866 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.049) 0:02:57.838 ********** 2020-03-09T00:20:05.904 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.037) 0:02:57.876 ********** 2020-03-09T00:20:05.940 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.036) 0:02:57.913 ********** 2020-03-09T00:20:05.944 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-09T00:20:05.946 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.948 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:05.987 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:05 +0000 (0:00:00.047) 0:02:57.960 ********** 2020-03-09T00:20:06.025 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.037) 0:02:57.998 ********** 2020-03-09T00:20:06.061 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.036) 0:02:58.034 ********** 2020-03-09T00:20:06.068 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:20:06.069 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:06.069 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.108 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.046) 0:02:58.080 ********** 2020-03-09T00:20:06.147 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.039) 0:02:58.119 ********** 2020-03-09T00:20:06.181 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.034) 0:02:58.154 ********** 2020-03-09T00:20:06.188 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-09T00:20:06.189 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.190 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.192 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.231 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.049) 0:02:58.204 ********** 2020-03-09T00:20:06.264 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.032) 0:02:58.237 ********** 2020-03-09T00:20:06.304 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.040) 0:02:58.277 ********** 2020-03-09T00:20:06.311 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-09T00:20:06.313 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-09T00:20:06.313 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.352 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.048) 0:02:58.325 ********** 2020-03-09T00:20:06.387 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.034) 0:02:58.360 ********** 2020-03-09T00:20:06.424 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.037) 0:02:58.397 ********** 2020-03-09T00:20:06.432 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.433 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.474 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.049) 0:02:58.446 ********** 2020-03-09T00:20:06.509 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.035) 0:02:58.482 ********** 2020-03-09T00:20:06.544 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.035) 0:02:58.517 ********** 2020-03-09T00:20:06.550 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:06.552 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.588 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.044) 0:02:58.561 ********** 2020-03-09T00:20:06.630 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.042) 0:02:58.603 ********** 2020-03-09T00:20:06.636 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-09T00:20:06.638 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.640 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.677 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.046) 0:02:58.649 ********** 2020-03-09T00:20:06.709 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.031) 0:02:58.681 ********** 2020-03-09T00:20:06.713 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:06.714 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-09T00:20:06.716 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set grub config.] ********************************************* 2020-03-09T00:20:06.716 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.753 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.044) 0:02:58.725 ********** 2020-03-09T00:20:06.784 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.031) 0:02:58.757 ********** 2020-03-09T00:20:06.815 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.030) 0:02:58.788 ********** 2020-03-09T00:20:06.819 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-09T00:20:06.821 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.859 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.044) 0:02:58.832 ********** 2020-03-09T00:20:06.894 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.034) 0:02:58.867 ********** 2020-03-09T00:20:06.928 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.033) 0:02:58.900 ********** 2020-03-09T00:20:06.932 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-09T00:20:06.934 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:06.979 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:06 +0000 (0:00:00.050) 0:02:58.951 ********** 2020-03-09T00:20:07.011 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.032) 0:02:58.984 ********** 2020-03-09T00:20:07.016 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-09T00:20:07.016 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.018 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.056 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.044) 0:02:59.028 ********** 2020-03-09T00:20:07.091 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.034) 0:02:59.063 ********** 2020-03-09T00:20:07.096 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:07.097 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:07.099 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-09T00:20:07.099 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.136 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.045) 0:02:59.109 ********** 2020-03-09T00:20:07.171 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.034) 0:02:59.144 ********** 2020-03-09T00:20:07.208 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.036) 0:02:59.180 ********** 2020-03-09T00:20:07.213 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-09T00:20:07.214 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.216 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.256 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.047) 0:02:59.228 ********** 2020-03-09T00:20:07.290 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.033) 0:02:59.262 ********** 2020-03-09T00:20:07.324 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.034) 0:02:59.297 ********** 2020-03-09T00:20:07.329 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-09T00:20:07.331 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Stop collectl] ************************************************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.369 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.044) 0:02:59.342 ********** 2020-03-09T00:20:07.404 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.035) 0:02:59.377 ********** 2020-03-09T00:20:07.412 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-09T00:20:07.413 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.414 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:07.416 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-09T00:20:07.454 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.049) 0:02:59.427 ********** 2020-03-09T00:20:07.492 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.037) 0:02:59.465 ********** 2020-03-09T00:20:07.546 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.053) 0:02:59.518 ********** 2020-03-09T00:20:07.552 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-09T00:20:07.553 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.554 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.554 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-09T00:20:07.555 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.603 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.057) 0:02:59.576 ********** 2020-03-09T00:20:07.637 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.033) 0:02:59.609 ********** 2020-03-09T00:20:07.674 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.036) 0:02:59.646 ********** 2020-03-09T00:20:07.679 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-09T00:20:07.681 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Stop collectl] ************************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.721 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.047) 0:02:59.694 ********** 2020-03-09T00:20:07.755 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.034) 0:02:59.728 ********** 2020-03-09T00:20:07.790 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.034) 0:02:59.763 ********** 2020-03-09T00:20:07.795 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-09T00:20:07.797 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-09T00:20:07.797 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.798 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:07.835 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.045) 0:02:59.808 ********** 2020-03-09T00:20:07.873 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.037) 0:02:59.846 ********** 2020-03-09T00:20:07.908 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.035) 0:02:59.881 ********** 2020-03-09T00:20:07.915 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-09T00:20:07.917 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:07.918 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-09T00:20:07.955 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.046) 0:02:59.928 ********** 2020-03-09T00:20:07.993 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:07 +0000 (0:00:00.037) 0:02:59.965 ********** 2020-03-09T00:20:08.030 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.036) 0:03:00.002 ********** 2020-03-09T00:20:08.037 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-09T00:20:08.039 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-09T00:20:08.039 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.041 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-09T00:20:08.083 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.052) 0:03:00.055 ********** 2020-03-09T00:20:08.117 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.034) 0:03:00.089 ********** 2020-03-09T00:20:08.124 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.125 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.126 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi104.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.127 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.129 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.130 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-09T00:20:08.168 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.051) 0:03:00.141 ********** 2020-03-09T00:20:08.219 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.050) 0:03:00.191 ********** 2020-03-09T00:20:08.274 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.055) 0:03:00.247 ********** 2020-03-09T00:20:08.281 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-09T00:20:08.283 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi050.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.284 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi050.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.286 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.286 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.287 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.333 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.058) 0:03:00.306 ********** 2020-03-09T00:20:08.375 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.041) 0:03:00.348 ********** 2020-03-09T00:20:08.383 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:08.385 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-09T00:20:08.385 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003352", "end": "2020-03-09 00:20:08.342667", "rc": 0, "start": "2020-03-09 00:20:08.339315", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-09T00:20:08.431 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.055) 0:03:00.404 ********** 2020-03-09T00:20:08.474 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.042) 0:03:00.446 ********** 2020-03-09T00:20:08.480 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-09T00:20:08.482 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.483 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi050.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003179", "end": "2020-03-09 00:20:08.450770", "rc": 0, "start": "2020-03-09 00:20:08.447591", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-09T00:20:08.528 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.055) 0:03:00.501 ********** 2020-03-09T00:20:08.568 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.039) 0:03:00.541 ********** 2020-03-09T00:20:08.570 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:08.601 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.032) 0:03:00.573 ********** 2020-03-09T00:20:08.607 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:08.608 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.609 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-09T00:20:08.611 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.653 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.052) 0:03:00.626 ********** 2020-03-09T00:20:08.692 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.038) 0:03:00.664 ********** 2020-03-09T00:20:08.728 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.036) 0:03:00.700 ********** 2020-03-09T00:20:08.733 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-09T00:20:08.733 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.735 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.773 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.045) 0:03:00.746 ********** 2020-03-09T00:20:08.807 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.033) 0:03:00.779 ********** 2020-03-09T00:20:08.846 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.039) 0:03:00.819 ********** 2020-03-09T00:20:08.852 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:08.853 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.854 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.894 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.047) 0:03:00.866 ********** 2020-03-09T00:20:08.929 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.034) 0:03:00.901 ********** 2020-03-09T00:20:08.962 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:08 +0000 (0:00:00.033) 0:03:00.934 ********** 2020-03-09T00:20:08.966 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-09T00:20:08.966 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:08.968 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.007 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.045) 0:03:00.980 ********** 2020-03-09T00:20:09.040 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.032) 0:03:01.013 ********** 2020-03-09T00:20:09.047 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:09.048 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.086 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.045) 0:03:01.059 ********** 2020-03-09T00:20:09.112 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:09.114 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.149 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.063) 0:03:01.122 ********** 2020-03-09T00:20:09.176 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:09.178 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.213 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.064) 0:03:01.186 ********** 2020-03-09T00:20:09.239 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-09T00:20:09.241 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.277 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.063) 0:03:01.249 ********** 2020-03-09T00:20:09.282 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:09.283 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-09T00:20:09.321 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.044) 0:03:01.294 ********** 2020-03-09T00:20:09.326 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-09T00:20:09.328 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.364 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.043) 0:03:01.337 ********** 2020-03-09T00:20:09.391 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-09T00:20:09.392 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.428 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.063) 0:03:01.400 ********** 2020-03-09T00:20:09.454 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-09T00:20:09.456 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.490 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.062) 0:03:01.463 ********** 2020-03-09T00:20:09.516 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-09T00:20:09.518 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.553 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.063) 0:03:01.526 ********** 2020-03-09T00:20:09.558 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:09.559 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:09.599 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.045) 0:03:01.571 ********** 2020-03-09T00:20:09.603 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-09T00:20:09.605 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.643 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.043) 0:03:01.615 ********** 2020-03-09T00:20:09.668 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-09T00:20:09.670 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.705 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.061) 0:03:01.677 ********** 2020-03-09T00:20:09.731 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-09T00:20:09.733 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.768 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.063) 0:03:01.740 ********** 2020-03-09T00:20:09.794 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-09T00:20:09.796 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.830 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.062) 0:03:01.803 ********** 2020-03-09T00:20:09.835 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:09.836 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-09T00:20:09.877 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.047) 0:03:01.850 ********** 2020-03-09T00:20:09.882 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-09T00:20:09.883 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-09T00:20:09.928 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.050) 0:03:01.900 ********** 2020-03-09T00:20:09.967 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:09 +0000 (0:00:00.038) 0:03:01.939 ********** 2020-03-09T00:20:09.974 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-09T00:20:09.975 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:09.977 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:09.977 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item={u'block_used': 12958826, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227136282, u'size_available': 930350211072, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989893, u'device': u'/dev/sda1', u'inode_used': 65083, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227136282, "block_size": 4096, "block_total": 240095108, "block_used": 12958826, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989893, "inode_total": 61054976, "inode_used": 65083, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930350211072, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}} 2020-03-09T00:20:10.015 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.048) 0:03:01.988 ********** 2020-03-09T00:20:10.048 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.032) 0:03:02.020 ********** 2020-03-09T00:20:10.053 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-09T00:20:10.055 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-09T00:20:10.093 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.045) 0:03:02.066 ********** 2020-03-09T00:20:10.127 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.034) 0:03:02.100 ********** 2020-03-09T00:20:10.133 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-09T00:20:10.135 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:10.135 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={u'block_used': 12958840, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227136268, u'size_available': 930350153728, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989889, u'device': u'/dev/sda1', u'inode_used': 65087, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227136268, "block_size": 4096, "block_total": 240095108, "block_used": 12958840, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989889, "inode_total": 61054976, "inode_used": 65087, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930350153728, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}} 2020-03-09T00:20:10.173 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.045) 0:03:02.145 ********** 2020-03-09T00:20:10.206 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.033) 0:03:02.179 ********** 2020-03-09T00:20:10.212 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-09T00:20:10.213 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.215 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-09T00:20:10.218 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005196", "end": "2020-03-09 00:20:10.191169", "rc": 0, "start": "2020-03-09 00:20:10.185973", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:10.255 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.049) 0:03:02.228 ********** 2020-03-09T00:20:10.297 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.041) 0:03:02.270 ********** 2020-03-09T00:20:10.303 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-09T00:20:10.304 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.305 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi104.front.sepia.ceph.com] => (item={u'block_used': 12958826, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227136282, u'size_available': 930350211072, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989893, u'device': u'/dev/sda1', u'inode_used': 65083, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227136282, "block_size": 4096, "block_total": 240095108, "block_used": 12958826, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989893, "inode_total": 61054976, "inode_used": 65083, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930350211072, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.345 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.047) 0:03:02.317 ********** 2020-03-09T00:20:10.380 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.035) 0:03:02.352 ********** 2020-03-09T00:20:10.385 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-09T00:20:10.387 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-09T00:20:10.387 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-09T00:20:10.389 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005346", "end": "2020-03-09 00:20:10.361228", "rc": 0, "start": "2020-03-09 00:20:10.355882", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:10.427 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.046) 0:03:02.399 ********** 2020-03-09T00:20:10.462 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.035) 0:03:02.435 ********** 2020-03-09T00:20:10.503 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.041) 0:03:02.476 ********** 2020-03-09T00:20:10.510 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:10.512 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi050.front.sepia.ceph.com] => (item={u'block_used': 12958840, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227136268, u'size_available': 930350153728, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989889, u'device': u'/dev/sda1', u'inode_used': 65087, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227136268, "block_size": 4096, "block_total": 240095108, "block_used": 12958840, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989889, "inode_total": 61054976, "inode_used": 65087, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930350153728, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.514 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:10.515 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QGJ7', u'wwn-0x5000c50091dfe2b2'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QGJ7', u'wwn': u'0x5000c50091dfe2b2', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QGJ7-part1', u'wwn-0x5000c50091dfe2b2-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QGJ7", "wwn-0x5000c50091dfe2b2"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QGJ7-part1", "wwn-0x5000c50091dfe2b2-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QGJ7", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:10.517 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50091dfe2b2"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.555 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.051) 0:03:02.528 ********** 2020-03-09T00:20:10.591 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.035) 0:03:02.564 ********** 2020-03-09T00:20:10.597 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:10.597 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-09T00:20:10.599 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-09T00:20:10.637 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.046) 0:03:02.610 ********** 2020-03-09T00:20:10.672 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.034) 0:03:02.645 ********** 2020-03-09T00:20:10.684 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:10.685 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.688 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:10.689 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HNFD', u'wwn-0x5000c500871ad2da'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HNFD', u'wwn': u'0x5000c500871ad2da', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1', u'wwn-0x5000c500871ad2da-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD", "wwn-0x5000c500871ad2da"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1", "wwn-0x5000c500871ad2da-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HNFD", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:10.691 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871ad2da"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.736 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.063) 0:03:02.708 ********** 2020-03-09T00:20:10.745 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-09T00:20:10.746 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.747 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi148.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.749 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.751 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.763 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:10.764 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi148.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.806 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.066) 0:03:02.775 ********** 2020-03-09T00:20:10.811 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.853 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.050) 0:03:02.825 ********** 2020-03-09T00:20:10.858 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-09T00:20:10.858 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.897 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.044) 0:03:02.870 ********** 2020-03-09T00:20:10.923 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-09T00:20:10.925 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.962 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:10 +0000 (0:00:00.064) 0:03:02.934 ********** 2020-03-09T00:20:10.989 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-09T00:20:10.992 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:10.993 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-09T00:20:10.995 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003830", "end": "2020-03-09 00:20:10.974527", "rc": 0, "start": "2020-03-09 00:20:10.970697", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-09T00:20:11.030 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.068) 0:03:03.003 ********** 2020-03-09T00:20:11.069 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.039) 0:03:03.042 ********** 2020-03-09T00:20:11.075 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-09T00:20:11.075 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.115 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.045) 0:03:03.088 ********** 2020-03-09T00:20:11.120 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-09T00:20:11.121 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.164 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.048) 0:03:03.136 ********** 2020-03-09T00:20:11.169 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-09T00:20:11.169 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.211 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.047) 0:03:03.184 ********** 2020-03-09T00:20:11.216 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-09T00:20:11.218 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.257 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.045) 0:03:03.230 ********** 2020-03-09T00:20:11.262 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-09T00:20:11.264 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.302 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.044) 0:03:03.274 ********** 2020-03-09T00:20:11.310 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-09T00:20:11.311 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.349 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.047) 0:03:03.322 ********** 2020-03-09T00:20:11.355 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-09T00:20:11.355 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.392 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.042) 0:03:03.364 ********** 2020-03-09T00:20:11.397 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-09T00:20:11.399 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.438 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.045) 0:03:03.410 ********** 2020-03-09T00:20:11.443 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-09T00:20:11.443 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.481 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.043) 0:03:03.454 ********** 2020-03-09T00:20:11.507 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-09T00:20:11.508 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.544 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.063) 0:03:03.517 ********** 2020-03-09T00:20:11.570 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-09T00:20:11.572 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.607 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.062) 0:03:03.579 ********** 2020-03-09T00:20:11.633 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-09T00:20:11.635 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.669 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.062) 0:03:03.642 ********** 2020-03-09T00:20:11.695 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-09T00:20:11.697 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.731 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.062) 0:03:03.704 ********** 2020-03-09T00:20:11.758 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-09T00:20:11.759 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.794 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.062) 0:03:03.767 ********** 2020-03-09T00:20:11.820 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-09T00:20:11.822 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.856 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.062) 0:03:03.829 ********** 2020-03-09T00:20:11.882 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-09T00:20:11.884 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.919 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.062) 0:03:03.892 ********** 2020-03-09T00:20:11.945 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-09T00:20:11.947 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:11.982 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:11 +0000 (0:00:00.062) 0:03:03.954 ********** 2020-03-09T00:20:12.018 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:12.020 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-09T00:20:12.056 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.074) 0:03:04.029 ********** 2020-03-09T00:20:12.104 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-09T00:20:12.105 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.134 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-09T00:20:12.135 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.135 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.147 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:12.148 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-zqe5gw-mqD6-m7jM-jRSX-R1v4-Ysbt-11nyGh', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400NW400BGN', u'nvme-nvme.8086-504846543632303430304e5734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.443605", "end": "2020-03-09 00:20:12.112651", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-zqe5gw-mqD6-m7jM-jRSX-R1v4-Ysbt-11nyGh", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NW400BGN", "nvme-nvme.8086-504846543632303430304e5734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:10.669046", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-09T00:20:12.149 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:12.149 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-09T00:20:12.150 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.151 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.161 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.203 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.146) 0:03:04.175 ********** 2020-03-09T00:20:12.345 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-09T00:20:12.345 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003820", "end": "2020-03-09 00:20:12.320109", "rc": 0, "start": "2020-03-09 00:20:12.316289", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-09T00:20:12.394 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.190) 0:03:04.366 ********** 2020-03-09T00:20:12.402 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-09T00:20:12.403 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-09T00:20:12.404 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:12.406 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Su3vJH-HdIy-cKJB-djvU-4not-7PL5-Q2fIzU', u'nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN', u'nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.494185", "end": "2020-03-09 00:20:12.355529", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-Su3vJH-HdIy-cKJB-djvU-4not-7PL5-Q2fIzU", "nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN", "nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:10.861344", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGP 2020-03-09T00:20:12.407 INFO:teuthology.task.ansible.out:T data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:12.447 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.053) 0:03:04.420 ********** 2020-03-09T00:20:12.453 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-09T00:20:12.453 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.499 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.052) 0:03:04.472 ********** 2020-03-09T00:20:12.506 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:12.507 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item={u'block_used': 12962518, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227132590, u'size_available': 930335088640, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989895, u'device': u'/dev/sda1', u'inode_used': 65081, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227132590, "block_size": 4096, "block_total": 240095108, "block_used": 12962518, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989895, "inode_total": 61054976, "inode_used": 65081, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930335088640, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}} 2020-03-09T00:20:12.546 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.047) 0:03:04.519 ********** 2020-03-09T00:20:12.551 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-09T00:20:12.551 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.591 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.044) 0:03:04.564 ********** 2020-03-09T00:20:12.618 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-09T00:20:12.619 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.656 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.064) 0:03:04.628 ********** 2020-03-09T00:20:12.682 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-09T00:20:12.684 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.719 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.062) 0:03:04.691 ********** 2020-03-09T00:20:12.755 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-09T00:20:12.757 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005066", "end": "2020-03-09 00:20:12.730172", "rc": 0, "start": "2020-03-09 00:20:12.725106", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:12.803 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.084) 0:03:04.776 ********** 2020-03-09T00:20:12.841 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-09T00:20:12.842 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'block_used': 12962518, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227132590, u'size_available': 930335088640, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989895, u'device': u'/dev/sda1', u'inode_used': 65081, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227132590, "block_size": 4096, "block_total": 240095108, "block_used": 12962518, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989895, "inode_total": 61054976, "inode_used": 65081, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930335088640, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:12.880 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.076) 0:03:04.853 ********** 2020-03-09T00:20:12.906 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-09T00:20:12.944 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:12 +0000 (0:00:00.063) 0:03:04.916 ********** 2020-03-09T00:20:12.985 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:12.987 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFK4', u'wwn-0x5000c50092924465'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFK4', u'wwn': u'0x5000c50092924465', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1', u'wwn-0x5000c50092924465-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4", "wwn-0x5000c50092924465"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1", "wwn-0x5000c50092924465-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFK4", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:12.988 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50092924465"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:13.409 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4ewlAFOWyRd8qeYxBKQakwmw3ONoepDwK'], u'uuids': [u'0ef57032-380c-4a87-b73c-a643516c4456']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.061820", "end": "2020-03-09 00:20:13.372559", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4ewlAFOWyRd8qeYxBKQakwmw3ONoepDwK"], "labels": [], "masters": [], "uuids": ["0ef57032-380c-4a87-b73c-a643516c4456"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:12.310739", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:13.652 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3H79Jq2SYpyFHOXUxjEOmnaPNt7cVAknCc'], u'uuids': [u'be28e69c-0e67-40d7-9d0a-ddaf8d4ccdea']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.051076", "end": "2020-03-09 00:20:13.617171", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3H79Jq2SYpyFHOXUxjEOmnaPNt7cVAknCc"], "labels": [], "masters": [], "uuids": ["be28e69c-0e67-40d7-9d0a-ddaf8d4ccdea"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:12.566095", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:13.666 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-09T00:20:13.668 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-09T00:20:13.703 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:13 +0000 (0:00:00.759) 0:03:05.676 ********** 2020-03-09T00:20:13.741 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:13.742 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={u'block_used': 12962517, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227132591, u'size_available': 930335092736, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989894, u'device': u'/dev/sda1', u'inode_used': 65082, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227132591, "block_size": 4096, "block_total": 240095108, "block_used": 12962517, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989894, "inode_total": 61054976, "inode_used": 65082, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930335092736, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}} 2020-03-09T00:20:13.783 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:13 +0000 (0:00:00.079) 0:03:05.755 ********** 2020-03-09T00:20:13.988 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-09T00:20:13.990 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005120", "end": "2020-03-09 00:20:13.960902", "rc": 0, "start": "2020-03-09 00:20:13.955782", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:14.029 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:14 +0000 (0:00:00.246) 0:03:06.002 ********** 2020-03-09T00:20:14.069 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-09T00:20:14.070 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'block_used': 12962517, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227132591, u'size_available': 930335092736, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989894, u'device': u'/dev/sda1', u'inode_used': 65082, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227132591, "block_size": 4096, "block_total": 240095108, "block_used": 12962517, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989894, "inode_total": 61054976, "inode_used": 65082, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930335092736, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:14.114 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:14 +0000 (0:00:00.085) 0:03:06.087 ********** 2020-03-09T00:20:14.141 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-09T00:20:14.181 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:14 +0000 (0:00:00.066) 0:03:06.154 ********** 2020-03-09T00:20:14.221 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:14.223 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFR4', u'wwn-0x5000c500929226ed'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFR4', u'wwn': u'0x5000c500929226ed', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1', u'wwn-0x5000c500929226ed-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4", "wwn-0x5000c500929226ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1", "wwn-0x5000c500929226ed-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFR4", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:14.224 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500929226ed"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:14.499 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-eMymow-0mxS-lt2L-dm7f-cfRD-Zfbr-AHrUdD', u'nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN', u'nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.312935", "end": "2020-03-09 00:20:14.462788", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-eMymow-0mxS-lt2L-dm7f-cfRD-Zfbr-AHrUdD", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN", "nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:13.149853", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-09T00:20:14.501 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:14.658 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4xogsa7QbPzxrK6Iefrrs8shULM63LOse'], u'uuids': [u'635b9bea-6091-4eba-8e87-e0b33444c41b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.050067", "end": "2020-03-09 00:20:14.625719", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4xogsa7QbPzxrK6Iefrrs8shULM63LOse"], "labels": [], "masters": [], "uuids": ["635b9bea-6091-4eba-8e87-e0b33444c41b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:13.575652", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:14.932 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HdncL2yJkOGn7JLYcX8Ab3i0zrDecfd32'], u'uuids': [u'1c13dedc-8e7e-46f2-aec2-7a22e3e497ac']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.065590", "end": "2020-03-09 00:20:14.894871", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HdncL2yJkOGn7JLYcX8Ab3i0zrDecfd32"], "labels": [], "masters": [], "uuids": ["1c13dedc-8e7e-46f2-aec2-7a22e3e497ac"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:13.829281", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:15.753 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2kYhmefPuPU1qNUNv96IZjvdsgQVumkGg'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.058121", "end": "2020-03-09 00:20:15.719460", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2kYhmefPuPU1qNUNv96IZjvdsgQVumkGg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:14.661339", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:15.903 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c40D1NLcjOD1ijx2jCZybB20VDBYOdfbIp'], u'uuids': [u'af75e9a1-d360-4e98-abba-8db6ee5fba75']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.052893", "end": "2020-03-09 00:20:15.872308", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c40D1NLcjOD1ijx2jCZybB20VDBYOdfbIp"], "labels": [], "masters": [], "uuids": ["af75e9a1-d360-4e98-abba-8db6ee5fba75"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:14.819415", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:16.040 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-6KjdTP-lAUg-zQv4-CeRl-nZEq-LjD0-d1n05y', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN', u'nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.625484", "end": "2020-03-09 00:20:16.012658", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-6KjdTP-lAUg-zQv4-CeRl-nZEq-LjD0-d1n05y", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN", "nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:14.387174", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-09T00:20:16.042 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:16.209 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3Hhrfa54tl0mzsZaiGc5VX8Fmym0st7MiH'], u'uuids': [u'1d3061b2-5073-45dd-9366-f56c50c55701']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.057196", "end": "2020-03-09 00:20:16.181942", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3Hhrfa54tl0mzsZaiGc5VX8Fmym0st7MiH"], "labels": [], "masters": [], "uuids": ["1d3061b2-5073-45dd-9366-f56c50c55701"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:15.124746", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:16.986 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2U6oTj4fmNqp35d0zl7K524uBe3c262sJ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.039282", "end": "2020-03-09 00:20:16.953506", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2U6oTj4fmNqp35d0zl7K524uBe3c262sJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:15.914224", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:17.165 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4VczVNTDo2fJHhmVc1gfPAungH2qwnfX4'], u'uuids': [u'56ed7064-2ab0-4be6-8222-dfb78f72c23e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.070442", "end": "2020-03-09 00:20:17.139420", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4VczVNTDo2fJHhmVc1gfPAungH2qwnfX4"], "labels": [], "masters": [], "uuids": ["56ed7064-2ab0-4be6-8222-dfb78f72c23e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:16.068978", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:17.292 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXvP4DeerV9VH535go9c5mRH8D1d5gmHPY'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.053065", "end": "2020-03-09 00:20:17.263485", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXvP4DeerV9VH535go9c5mRH8D1d5gmHPY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:16.210420", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:17.482 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HYPJwAU36X7gDWYQ0Mj3icbbP5tIg8hc7'], u'uuids': [u'1eeffb89-5be6-4224-9ca8-eb8c62bc1bf3']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.053441", "end": "2020-03-09 00:20:17.449887", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HYPJwAU36X7gDWYQ0Mj3icbbP5tIg8hc7"], "labels": [], "masters": [], "uuids": ["1eeffb89-5be6-4224-9ca8-eb8c62bc1bf3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:16.396446", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:18.258 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2X1dFVsBcerDqzgB600iOh0c7KUOJDNYq'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.070787", "end": "2020-03-09 00:20:18.220264", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2X1dFVsBcerDqzgB600iOh0c7KUOJDNYq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:17.149477", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:18.425 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4cMb3nnkow6p3lwfkjfBA7EbCr9tYXrvl'], u'uuids': [u'a44a07e0-1c5c-477b-951f-3518351b085c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.045207", "end": "2020-03-09 00:20:18.381724", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4cMb3nnkow6p3lwfkjfBA7EbCr9tYXrvl"], "labels": [], "masters": [], "uuids": ["a44a07e0-1c5c-477b-951f-3518351b085c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:17.336517", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:18.599 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:18 +0000 (0:00:04.417) 0:03:10.571 ********** 2020-03-09T00:20:18.619 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXp5ZvV8B8pngPACrZOpayLZdODtbZRyrV'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.041076", "end": "2020-03-09 00:20:18.511983", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXp5ZvV8B8pngPACrZOpayLZdODtbZRyrV"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:17.470907", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:18.672 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:18.674 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QGJ7', u'wwn-0x5000c50091dfe2b2'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QGJ7', u'wwn': u'0x5000c50091dfe2b2', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QGJ7-part1', u'wwn-0x5000c50091dfe2b2-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QGJ7", "wwn-0x5000c50091dfe2b2"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QGJ7-part1", "wwn-0x5000c50091dfe2b2-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QGJ7", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:18.675 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50091dfe2b2"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:18.909 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-zqe5gw-mqD6-m7jM-jRSX-R1v4-Ysbt-11nyGh', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400NW400BGN', u'nvme-nvme.8086-504846543632303430304e5734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037379", "end": "2020-03-09 00:20:18.873946", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-zqe5gw-mqD6-m7jM-jRSX-R1v4-Ysbt-11nyGh", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NW400BGN", "nvme-nvme.8086-504846543632303430304e5734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:18.836567", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-09T00:20:18.910 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.03471 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.03471 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:18.922 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:18.923 INFO:teuthology.task.ansible.out:TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HHWin5c2DbNB01haeHzFuCtL2KFsbHBKc'], u'uuids': [u'0c6383e3-a378-40b8-bb8f-6c2e4968f07a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.210687", "end": "2020-03-09 00:20:18.878806", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HHWin5c2DbNB01haeHzFuCtL2KFsbHBKc"], "labels": [], "masters": [], "uuids": ["0c6383e3-a378-40b8-bb8f-6c2e4968f07a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:17.668119", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! Yo 2020-03-09T00:20:18.925 INFO:teuthology.task.ansible.out:u may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:19.006 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:19 +0000 (0:00:00.406) 0:03:10.978 ********** 2020-03-09T00:20:19.068 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:19.069 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HNFD', u'wwn-0x5000c500871ad2da'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HNFD', u'wwn': u'0x5000c500871ad2da', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1', u'wwn-0x5000c500871ad2da-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD", "wwn-0x5000c500871ad2da"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1", "wwn-0x5000c500871ad2da-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HNFD", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:19.071 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871ad2da"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:19.242 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4ewlAFOWyRd8qeYxBKQakwmw3ONoepDwK'], u'uuids': [u'0ef57032-380c-4a87-b73c-a643516c4456']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.141237", "end": "2020-03-09 00:20:19.220089", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4ewlAFOWyRd8qeYxBKQakwmw3ONoepDwK"], "labels": [], "masters": [], "uuids": ["0ef57032-380c-4a87-b73c-a643516c4456"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:19.078852", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138673 s, 832 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138673 s, 832 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:19.308 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Su3vJH-HdIy-cKJB-djvU-4not-7PL5-Q2fIzU', u'nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN', u'nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036964", "end": "2020-03-09 00:20:19.282987", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-Su3vJH-HdIy-cKJB-djvU-4not-7PL5-Q2fIzU", "nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN", "nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:19.246023", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 2020-03-09T00:20:19.310 INFO:teuthology.task.ansible.out: 0.0340069 s, 3.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0340069 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:19.580 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4xogsa7QbPzxrK6Iefrrs8shULM63LOse'], u'uuids': [u'635b9bea-6091-4eba-8e87-e0b33444c41b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141985", "end": "2020-03-09 00:20:19.547709", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4xogsa7QbPzxrK6Iefrrs8shULM63LOse"], "labels": [], "masters": [], "uuids": ["635b9bea-6091-4eba-8e87-e0b33444c41b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:19.405724", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139265 s, 828 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139265 s, 828 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:19.604 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:19.605 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP203vKscGEncpFf3dUVfQ5dk30Cw2qzusg'], u'uuids': [u'8f38917b-96e5-4d81-8847-ab88fe3445c6']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.046807", "end": "2020-03-09 00:20:19.570744", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP203vKscGEncpFf3dUVfQ5dk30Cw2qzusg"], "labels": [], "masters": [], "uuids": ["8f38917b-96e5-4d81-8847-ab88fe3445c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:18.523937", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:19.607 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:19.607 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3H79Jq2SYpyFHOXUxjEOmnaPNt7cVAknCc'], u'uuids': [u'be28e69c-0e67-40d7-9d0a-ddaf8d4ccdea']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.095026", "end": "2020-03-09 00:20:19.580731", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3H79Jq2SYpyFHOXUxjEOmnaPNt7cVAknCc"], "labels": [], "masters": [], "uuids": ["be28e69c-0e67-40d7-9d0a-ddaf8d4ccdea"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:19.485705", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0923468 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0923468 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:19.864 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:19.865 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXmG1lkUdmA7etdgfze3PJiYx1IuI8GMWI'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.046564", "end": "2020-03-09 00:20:19.838213", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXmG1lkUdmA7etdgfze3PJiYx1IuI8GMWI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:18.791649", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:19.900 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:19.901 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HdncL2yJkOGn7JLYcX8Ab3i0zrDecfd32'], u'uuids': [u'1c13dedc-8e7e-46f2-aec2-7a22e3e497ac']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.095028", "end": "2020-03-09 00:20:19.873238", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HdncL2yJkOGn7JLYcX8Ab3i0zrDecfd32"], "labels": [], "masters": [], "uuids": ["1c13dedc-8e7e-46f2-aec2-7a22e3e497ac"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:19.778210", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0923141 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0923141 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:19.914 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c40D1NLcjOD1ijx2jCZybB20VDBYOdfbIp'], u'uuids': [u'af75e9a1-d360-4e98-abba-8db6ee5fba75']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.141154", "end": "2020-03-09 00:20:19.885542", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c40D1NLcjOD1ijx2jCZybB20VDBYOdfbIp"], "labels": [], "masters": [], "uuids": ["af75e9a1-d360-4e98-abba-8db6ee5fba75"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:19.744388", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13851 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13851 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:20.199 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3Hhrfa54tl0mzsZaiGc5VX8Fmym0st7MiH'], u'uuids': [u'1d3061b2-5073-45dd-9366-f56c50c55701']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.094481", "end": "2020-03-09 00:20:20.169055", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3Hhrfa54tl0mzsZaiGc5VX8Fmym0st7MiH"], "labels": [], "masters": [], "uuids": ["1d3061b2-5073-45dd-9366-f56c50c55701"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:20.074574", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.091768 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.091768 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:20.243 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4VczVNTDo2fJHhmVc1gfPAungH2qwnfX4'], u'uuids': [u'56ed7064-2ab0-4be6-8222-dfb78f72c23e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.141226", "end": "2020-03-09 00:20:20.213895", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4VczVNTDo2fJHhmVc1gfPAungH2qwnfX4"], "labels": [], "masters": [], "uuids": ["56ed7064-2ab0-4be6-8222-dfb78f72c23e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:20.072669", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138497 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138497 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:20.489 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HYPJwAU36X7gDWYQ0Mj3icbbP5tIg8hc7'], u'uuids': [u'1eeffb89-5be6-4224-9ca8-eb8c62bc1bf3']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.094986", "end": "2020-03-09 00:20:20.463319", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HYPJwAU36X7gDWYQ0Mj3icbbP5tIg8hc7"], "labels": [], "masters": [], "uuids": ["1eeffb89-5be6-4224-9ca8-eb8c62bc1bf3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:20.368333", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.092307 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.092307 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:20.578 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4cMb3nnkow6p3lwfkjfBA7EbCr9tYXrvl'], u'uuids': [u'a44a07e0-1c5c-477b-951f-3518351b085c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139332", "end": "2020-03-09 00:20:20.547374", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-2mnyEnpwngdmKNfHoP2mtoTqQs06V1c4cMb3nnkow6p3lwfkjfBA7EbCr9tYXrvl"], "labels": [], "masters": [], "uuids": ["a44a07e0-1c5c-477b-951f-3518351b085c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:20.408042", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.1367 s, 844 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.1367 s, 844 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:20.632 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:20 +0000 (0:00:01.626) 0:03:12.605 ********** 2020-03-09T00:20:20.780 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HHWin5c2DbNB01haeHzFuCtL2KFsbHBKc'], u'uuids': [u'0c6383e3-a378-40b8-bb8f-6c2e4968f07a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.094892", "end": "2020-03-09 00:20:20.755599", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-D99jLfn7ieavuMQfUHhw9khIXsMapf3HHWin5c2DbNB01haeHzFuCtL2KFsbHBKc"], "labels": [], "masters": [], "uuids": ["0c6383e3-a378-40b8-bb8f-6c2e4968f07a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:20.660707", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0922339 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0922339 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:20.851 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:20 +0000 (0:00:00.219) 0:03:12.824 ********** 2020-03-09T00:20:20.886 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:20.888 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP247tezkmz4bMkRBcryBlf7wVyPg46ZzEI'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.079291", "end": "2020-03-09 00:20:20.848687", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP247tezkmz4bMkRBcryBlf7wVyPg46ZzEI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:19.769396", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:20.949 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:20 +0000 (0:00:00.097) 0:03:12.922 ********** 2020-03-09T00:20:20.957 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-09T00:20:20.958 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.068118", "end": "2020-03-09 00:20:20.892337", "rc": 0, "start": "2020-03-09 00:20:20.824219", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:21.002 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:20 +0000 (0:00:00.053) 0:03:12.975 ********** 2020-03-09T00:20:21.010 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:21.012 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFK4', u'wwn-0x5000c50092924465'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFK4', u'wwn': u'0x5000c50092924465', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1', u'wwn-0x5000c50092924465-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4", "wwn-0x5000c50092924465"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFK4-part1", "wwn-0x5000c50092924465-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFK4", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:21.014 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50092924465"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.131 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:21.133 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFX1VLdzggcFVTdWfjdeZmegEwB3PLdVMHk'], u'uuids': [u'1aec7345-86c8-4e6b-9388-f18f15770b6a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.063134", "end": "2020-03-09 00:20:21.097678", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFX1VLdzggcFVTdWfjdeZmegEwB3PLdVMHk"], "labels": [], "masters": [], "uuids": ["1aec7345-86c8-4e6b-9388-f18f15770b6a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:20.034544", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:21.166 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-09T00:20:21.168 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.066688", "end": "2020-03-09 00:20:21.129195", "rc": 0, "start": "2020-03-09 00:20:21.062507", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:21.212 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.209) 0:03:13.184 ********** 2020-03-09T00:20:21.271 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:21.274 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-eMymow-0mxS-lt2L-dm7f-cfRD-Zfbr-AHrUdD', u'nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN', u'nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037502", "end": "2020-03-09 00:20:21.234509", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-eMymow-0mxS-lt2L-dm7f-cfRD-Zfbr-AHrUdD", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005G400BGN", "nvme-nvme.8086-43564654363233333030354734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:21.197007", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-09T00:20:21.276 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0348138 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0348138 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} TASK [testnode : Check for physical volumes] *********************************** 2020-03-09T00:20:21.276 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.063533", "end": "2020-03-09 00:20:21.250517", "rc": 0, "start": "2020-03-09 00:20:21.186984", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:21.315 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.103) 0:03:13.288 ********** 2020-03-09T00:20:21.365 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-09T00:20:21.408 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.092) 0:03:13.381 ********** 2020-03-09T00:20:21.436 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-09T00:20:21.438 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.480 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.072) 0:03:13.453 ********** 2020-03-09T00:20:21.488 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-09T00:20:21.490 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.047273", "end": "2020-03-09 00:20:21.451065", "rc": 0, "start": "2020-03-09 00:20:21.403792", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:21.529 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.048) 0:03:13.501 ********** 2020-03-09T00:20:21.536 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-09T00:20:21.537 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.579 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.049) 0:03:13.551 ********** 2020-03-09T00:20:21.585 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-09T00:20:21.631 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.052) 0:03:13.603 ********** 2020-03-09T00:20:21.638 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:21.639 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2kYhmefPuPU1qNUNv96IZjvdsgQVumkGg'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140015", "end": "2020-03-09 00:20:21.576992", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2kYhmefPuPU1qNUNv96IZjvdsgQVumkGg"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:21.436977", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137359 s, 840 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137359 s, 840 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:21.641 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.685 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.053) 0:03:13.657 ********** 2020-03-09T00:20:21.690 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-09T00:20:21.692 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.741 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.057) 0:03:13.714 ********** 2020-03-09T00:20:21.749 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-09T00:20:21.751 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.789 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.047) 0:03:13.762 ********** 2020-03-09T00:20:21.796 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-09T00:20:21.798 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.838 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.048) 0:03:13.811 ********** 2020-03-09T00:20:21.866 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-09T00:20:21.868 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.906 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.068) 0:03:13.879 ********** 2020-03-09T00:20:21.935 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:21.937 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2U6oTj4fmNqp35d0zl7K524uBe3c262sJ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.139041", "end": "2020-03-09 00:20:21.909805", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2U6oTj4fmNqp35d0zl7K524uBe3c262sJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": 2020-03-09T00:20:21.939 INFO:teuthology.task.ansible.out:"0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:21.770764", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136449 s, 845 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136449 s, 845 MB/s"], "stdout": "", "stdout_lines": []} TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:21.975 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:21 +0000 (0:00:00.068) 0:03:13.948 ********** 2020-03-09T00:20:22.003 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-09T00:20:22.005 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:20:22.043 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.067) 0:03:14.015 ********** 2020-03-09T00:20:22.081 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:22.082 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={u'block_used': 12958826, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227136282, u'size_available': 930350211072, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989893, u'device': u'/dev/sda1', u'inode_used': 65083, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227136282, "block_size": 4096, "block_total": 240095108, "block_used": 12958826, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989893, "inode_total": 61054976, "inode_used": 65083, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930350211072, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.119 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.076) 0:03:14.092 ********** 2020-03-09T00:20:22.146 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-09T00:20:22.148 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.187 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.068) 0:03:14.160 ********** 2020-03-09T00:20:22.194 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-09T00:20:22.195 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:20:22.233 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.045) 0:03:14.206 ********** 2020-03-09T00:20:22.244 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.283 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.049) 0:03:14.255 ********** 2020-03-09T00:20:22.290 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:22.291 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2X1dFVsBcerDqzgB600iOh0c7KUOJDNYq'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.139608", "end": "2020-03-09 00:20:22.240304", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP2X1dFVsBcerDqzgB600iOh0c7KUOJDNYq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:22.100696", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136959 s, 842 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136959 s, 842 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:22.304 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:22.305 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={u'block_used': 12958840, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227136268, u'size_available': 930350153728, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989889, u'device': u'/dev/sda1', u'inode_used': 65087, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227136268, "block_size": 4096, "block_total": 240095108, "block_used": 12958840, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989889, "inode_total": 61054976, "inode_used": 65087, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930350153728, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.307 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.343 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.060) 0:03:14.316 ********** 2020-03-09T00:20:22.384 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.041) 0:03:14.357 ********** 2020-03-09T00:20:22.391 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-09T00:20:22.392 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.430 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.045) 0:03:14.403 ********** 2020-03-09T00:20:22.440 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-09T00:20:22.441 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.446 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-09T00:20:22.448 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXkotuwYBG1FICawDG9qSTv9dfJg2R6Jxh'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.075455", "end": "2020-03-09 00:20:22.408782", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXkotuwYBG1FICawDG9qSTv9dfJg2R6Jxh"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:21.333327", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-09T00:20:22.518 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.087) 0:03:14.490 ********** 2020-03-09T00:20:22.553 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.035) 0:03:14.525 ********** 2020-03-09T00:20:22.560 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:22.562 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.603 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.050) 0:03:14.576 ********** 2020-03-09T00:20:22.643 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.039) 0:03:14.615 ********** 2020-03-09T00:20:22.650 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:22.652 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFR4', u'wwn-0x5000c500929226ed'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFR4', u'wwn': u'0x5000c500929226ed', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1', u'wwn-0x5000c500929226ed-part1'], u'uuids': [u'66e056a8-b246-4f81-96dd-96b23573000c']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4", "wwn-0x5000c500929226ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1", "wwn-0x5000c500929226ed-part1"], "labels": [], "masters": [], "uuids": ["66e056a8-b246-4f81-96dd-96b23573000c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFR4", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-09T00:20:22.654 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500929226ed"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.656 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP203vKscGEncpFf3dUVfQ5dk30Cw2qzusg'], u'uuids': [u'8f38917b-96e5-4d81-8847-ab88fe3445c6']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.140860", "end": "2020-03-09 00:20:22.571983", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP203vKscGEncpFf3dUVfQ5dk30Cw2qzusg"], "labels": [], "masters": [], "uuids": ["8f38917b-96e5-4d81-8847-ab88fe3445c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:22.431123", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138291 s, 834 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138291 s, 834 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:22.656 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.695 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.052) 0:03:14.668 ********** 2020-03-09T00:20:22.700 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-09T00:20:22.702 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.740 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.044) 0:03:14.712 ********** 2020-03-09T00:20:22.747 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-09T00:20:22.749 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.785 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.044) 0:03:14.757 ********** 2020-03-09T00:20:22.812 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-09T00:20:22.814 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.851 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.066) 0:03:14.824 ********** 2020-03-09T00:20:22.858 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:22.860 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-6KjdTP-lAUg-zQv4-CeRl-nZEq-LjD0-d1n05y', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN', u'nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037790", "end": "2020-03-09 00:20:22.813378", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-6KjdTP-lAUg-zQv4-CeRl-nZEq-LjD0-d1n05y", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN", "nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:22.775588", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-09T00:20:22.861 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0349621 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0349621 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:22.882 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-09T00:20:22.884 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:22.919 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.068) 0:03:14.892 ********** 2020-03-09T00:20:22.937 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:22.938 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP247tezkmz4bMkRBcryBlf7wVyPg46ZzEI'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.136814", "end": "2020-03-09 00:20:22.907347", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-bRxT1oWXImSnbt9Q9wOdiCGpYi4gdoP247tezkmz4bMkRBcryBlf7wVyPg46ZzEI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:22.770533", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.134168 s, 860 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.134168 s, 860 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:22.983 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:22 +0000 (0:00:00.063) 0:03:14.955 ********** 2020-03-09T00:20:23.170 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXvP4DeerV9VH535go9c5mRH8D1d5gmHPY'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.138273", "end": "2020-03-09 00:20:23.138351", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXvP4DeerV9VH535go9c5mRH8D1d5gmHPY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:23.000078", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135844 s, 849 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135844 s, 849 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:23.267 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-09T00:20:23.269 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.077304", "end": "2020-03-09 00:20:23.233136", "rc": 0, "start": "2020-03-09 00:20:23.155832", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:23.304 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.321) 0:03:15.277 ********** 2020-03-09T00:20:23.501 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-09T00:20:23.502 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:23.504 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXp5ZvV8B8pngPACrZOpayLZdODtbZRyrV'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.149187", "end": "2020-03-09 00:20:23.476073", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXp5ZvV8B8pngPACrZOpayLZdODtbZRyrV"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:23.326886", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14669 s, 786 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14669 s, 786 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:23.506 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-09T00:20:23.506 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-09T00:20:23.545 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.240) 0:03:15.518 ********** 2020-03-09T00:20:23.580 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.035) 0:03:15.553 ********** 2020-03-09T00:20:23.587 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-09T00:20:23.588 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.050716", "end": "2020-03-09 00:20:23.529966", "rc": 0, "start": "2020-03-09 00:20:23.479250", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:23.629 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.048) 0:03:15.601 ********** 2020-03-09T00:20:23.656 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-09T00:20:23.699 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.069) 0:03:15.671 ********** 2020-03-09T00:20:23.726 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-09T00:20:23.727 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:23.768 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.069) 0:03:15.741 ********** 2020-03-09T00:20:23.795 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-09T00:20:23.797 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:23.836 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.067) 0:03:15.809 ********** 2020-03-09T00:20:23.842 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:23.843 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXmG1lkUdmA7etdgfze3PJiYx1IuI8GMWI'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.148453", "end": "2020-03-09 00:20:23.810326", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXmG1lkUdmA7etdgfze3PJiYx1IuI8GMWI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:23.661873", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145787 s, 791 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145787 s, 791 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:23.867 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-09T00:20:23.869 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:23.909 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.072) 0:03:15.881 ********** 2020-03-09T00:20:23.935 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-09T00:20:23.937 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:23.975 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:23 +0000 (0:00:00.066) 0:03:15.947 ********** 2020-03-09T00:20:23.976 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:24.180 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFX1VLdzggcFVTdWfjdeZmegEwB3PLdVMHk'], u'uuids': [u'1aec7345-86c8-4e6b-9388-f18f15770b6a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.140278", "end": "2020-03-09 00:20:24.151408", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFX1VLdzggcFVTdWfjdeZmegEwB3PLdVMHk"], "labels": [], "masters": [], "uuids": ["1aec7345-86c8-4e6b-9388-f18f15770b6a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:24.011130", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137651 s, 838 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137651 s, 838 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:24.230 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.254) 0:03:16.202 ********** 2020-03-09T00:20:24.248 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:24.249 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:24.262 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:24.274 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:24.276 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={u'block_used': 12962518, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227132590, u'size_available': 930335088640, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989895, u'device': u'/dev/sda1', u'inode_used': 65081, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227132590, "block_size": 4096, "block_total": 240095108, "block_used": 12962518, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989895, "inode_total": 61054976, "inode_used": 65081, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930335088640, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:24.316 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.086) 0:03:16.289 ********** 2020-03-09T00:20:24.356 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-09T00:20:24.358 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:24.398 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.081) 0:03:16.371 ********** 2020-03-09T00:20:24.459 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:24.460 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:24.500 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.101) 0:03:16.472 ********** 2020-03-09T00:20:24.517 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-09T00:20:24.519 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXkotuwYBG1FICawDG9qSTv9dfJg2R6Jxh'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.137613", "end": "2020-03-09 00:20:24.480906", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-JQJdWbSLe8s8ir2PG3ZLYGUr0ZXQBUFXkotuwYBG1FICawDG9qSTv9dfJg2R6Jxh"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-09 00:20:24.343293", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135221 s, 853 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135221 s, 853 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:24.530 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:24.532 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:24.568 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.068) 0:03:16.541 ********** 2020-03-09T00:20:24.612 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.043) 0:03:16.584 ********** 2020-03-09T00:20:24.618 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:24.620 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:24.645 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-09T00:20:24.647 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:24.684 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.071) 0:03:16.656 ********** 2020-03-09T00:20:24.714 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-09T00:20:24.715 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:24.753 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.069) 0:03:16.725 ********** 2020-03-09T00:20:24.790 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-09T00:20:24.792 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:24.831 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.078) 0:03:16.804 ********** 2020-03-09T00:20:24.838 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-09T00:20:24.839 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.063425", "end": "2020-03-09 00:20:24.802459", "rc": 0, "start": "2020-03-09 00:20:24.739034", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:24.878 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:24 +0000 (0:00:00.046) 0:03:16.850 ********** 2020-03-09T00:20:24.961 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:24.963 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:24.974 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:25.124 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-09T00:20:25.126 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049897", "end": "2020-03-09 00:20:25.097046", "rc": 0, "start": "2020-03-09 00:20:25.047149", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:25.181 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.303) 0:03:17.154 ********** 2020-03-09T00:20:25.202 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-09T00:20:25.250 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.068) 0:03:17.222 ********** 2020-03-09T00:20:25.288 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-09T00:20:25.290 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-09T00:20:25.290 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:25.357 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.107) 0:03:17.329 ********** 2020-03-09T00:20:25.405 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.048) 0:03:17.378 ********** 2020-03-09T00:20:25.411 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:25.412 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:25.414 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:25.414 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-09T00:20:25.414 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:25.457 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.051) 0:03:17.429 ********** 2020-03-09T00:20:25.484 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-09T00:20:25.486 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:25.536 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.079) 0:03:17.509 ********** 2020-03-09T00:20:25.563 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-09T00:20:25.565 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:25.601 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.065) 0:03:17.574 ********** 2020-03-09T00:20:25.683 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:25.685 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:25.751 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.149) 0:03:17.723 ********** 2020-03-09T00:20:25.758 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:25.800 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.049) 0:03:17.773 ********** 2020-03-09T00:20:25.818 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-09T00:20:25.820 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-09T00:20:25.863 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.062) 0:03:17.835 ********** 2020-03-09T00:20:25.869 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:25.870 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:25.925 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-09T00:20:25.927 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={u'block_used': 12962517, u'uuid': u'66e056a8-b246-4f81-96dd-96b23573000c', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227132591, u'size_available': 930335092736, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989894, u'device': u'/dev/sda1', u'inode_used': 65082, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227132591, "block_size": 4096, "block_total": 240095108, "block_used": 12962517, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989894, "inode_total": 61054976, "inode_used": 65082, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930335092736, "size_total": 983429562368, "uuid": "66e056a8-b246-4f81-96dd-96b23573000c"}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:25.965 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:25 +0000 (0:00:00.102) 0:03:17.938 ********** 2020-03-09T00:20:25.973 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-09T00:20:25.975 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-09T00:20:26.013 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.047) 0:03:17.985 ********** 2020-03-09T00:20:26.020 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-09T00:20:26.022 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.063 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.050) 0:03:18.036 ********** 2020-03-09T00:20:26.072 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-09T00:20:26.073 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-09T00:20:26.074 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** skipping: [smithi104.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.115 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.051) 0:03:18.088 ********** 2020-03-09T00:20:26.122 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:26.124 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.173 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.057) 0:03:18.146 ********** 2020-03-09T00:20:26.180 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:26.182 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.215 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:26.217 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:26.227 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:26.229 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.296 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.122) 0:03:18.269 ********** 2020-03-09T00:20:26.303 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:26.304 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:26.327 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.367 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.069) 0:03:18.339 ********** 2020-03-09T00:20:26.383 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:26.384 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:26.396 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-09T00:20:26.398 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.435 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.068) 0:03:18.408 ********** 2020-03-09T00:20:26.474 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:26.475 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:26.487 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:26.526 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.091) 0:03:18.499 ********** 2020-03-09T00:20:26.544 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:26.546 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:26.590 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:26.591 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:26.667 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:26.787 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:26.857 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:26.901 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:26 +0000 (0:00:00.374) 0:03:18.874 ********** 2020-03-09T00:20:26.908 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:26.909 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:26.974 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-09T00:20:26.976 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-09T00:20:27.015 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.113) 0:03:18.988 ********** 2020-03-09T00:20:27.023 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:27.024 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:27.077 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.062) 0:03:19.050 ********** 2020-03-09T00:20:27.108 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:27.148 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.071) 0:03:19.121 ********** 2020-03-09T00:20:27.187 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-09T00:20:27.189 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-09T00:20:27.230 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.082) 0:03:19.203 ********** 2020-03-09T00:20:27.237 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:27.238 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:27.281 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.050) 0:03:19.253 ********** 2020-03-09T00:20:27.311 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-09T00:20:27.312 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:27.351 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.070) 0:03:19.324 ********** 2020-03-09T00:20:27.390 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-09T00:20:27.392 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:27.434 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.083) 0:03:19.407 ********** 2020-03-09T00:20:27.494 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-09T00:20:27.497 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-09T00:20:27.535 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.100) 0:03:19.508 ********** 2020-03-09T00:20:27.579 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:27.581 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:27.747 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:27.749 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:27.769 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:27.770 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:27.781 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-09T00:20:27.783 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true} 2020-03-09T00:20:27.821 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.285) 0:03:19.794 ********** 2020-03-09T00:20:27.839 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true} 2020-03-09T00:20:27.877 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:27 +0000 (0:00:00.055) 0:03:19.850 ********** 2020-03-09T00:20:27.947 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:27.948 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:28.076 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:28.078 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:28.130 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:28.131 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:28.337 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:28.379 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.501) 0:03:20.351 ********** 2020-03-09T00:20:28.439 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-09T00:20:28.441 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-09T00:20:28.489 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.109) 0:03:20.461 ********** 2020-03-09T00:20:28.494 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:28.496 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:28.518 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-09T00:20:28.520 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:28.560 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.071) 0:03:20.533 ********** 2020-03-09T00:20:28.588 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-09T00:20:28.588 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:28.590 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:28.627 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.066) 0:03:20.599 ********** 2020-03-09T00:20:28.661 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.033) 0:03:20.633 ********** 2020-03-09T00:20:28.665 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-09T00:20:28.665 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:28.715 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.049) 0:03:20.683 ********** 2020-03-09T00:20:28.720 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:28.722 INFO:teuthology.task.ansible.out:TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-09T00:20:28.759 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.048) 0:03:20.732 ********** 2020-03-09T00:20:28.765 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-09T00:20:28.765 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:28.807 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.047) 0:03:20.780 ********** 2020-03-09T00:20:28.829 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-09T00:20:28.831 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-09T00:20:28.836 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-09T00:20:28.838 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:28.874 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.067) 0:03:20.847 ********** 2020-03-09T00:20:28.911 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.036) 0:03:20.884 ********** 2020-03-09T00:20:28.918 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:28.966 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:28 +0000 (0:00:00.054) 0:03:20.938 ********** 2020-03-09T00:20:28.993 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-09T00:20:28.995 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.031 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.065) 0:03:21.004 ********** 2020-03-09T00:20:29.048 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-09T00:20:29.050 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true} 2020-03-09T00:20:29.088 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.056) 0:03:21.061 ********** 2020-03-09T00:20:29.094 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-09T00:20:29.095 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.136 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.047) 0:03:21.108 ********** 2020-03-09T00:20:29.142 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-09T00:20:29.143 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-09T00:20:29.182 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.046) 0:03:21.155 ********** 2020-03-09T00:20:29.188 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-09T00:20:29.189 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.228 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.045) 0:03:21.200 ********** 2020-03-09T00:20:29.234 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:29.235 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.257 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-09T00:20:29.259 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.297 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.068) 0:03:21.269 ********** 2020-03-09T00:20:29.325 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-09T00:20:29.329 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.364 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.066) 0:03:21.336 ********** 2020-03-09T00:20:29.402 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-09T00:20:29.404 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-09T00:20:29.442 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.078) 0:03:21.415 ********** 2020-03-09T00:20:29.450 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:29.451 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:29.453 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-09T00:20:29.494 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.049) 0:03:21.465 ********** 2020-03-09T00:20:29.500 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-09T00:20:29.503 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.541 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.047) 0:03:21.512 ********** 2020-03-09T00:20:29.546 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.586 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.046) 0:03:21.559 ********** 2020-03-09T00:20:29.593 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-09T00:20:29.595 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.632 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.045) 0:03:21.605 ********** 2020-03-09T00:20:29.639 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:29.641 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.679 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.046) 0:03:21.652 ********** 2020-03-09T00:20:29.686 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-09T00:20:29.687 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.724 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.045) 0:03:21.697 ********** 2020-03-09T00:20:29.730 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.769 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.044) 0:03:21.742 ********** 2020-03-09T00:20:29.778 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-09T00:20:29.780 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.816 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.046) 0:03:21.788 ********** 2020-03-09T00:20:29.823 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:29.825 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.863 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.047) 0:03:21.836 ********** 2020-03-09T00:20:29.869 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-09T00:20:29.871 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.909 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.045) 0:03:21.882 ********** 2020-03-09T00:20:29.915 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:29.954 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.045) 0:03:21.927 ********** 2020-03-09T00:20:29.961 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-09T00:20:29.962 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-09T00:20:29.964 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.001 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:29 +0000 (0:00:00.046) 0:03:21.973 ********** 2020-03-09T00:20:30.034 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.033) 0:03:22.007 ********** 2020-03-09T00:20:30.040 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-09T00:20:30.041 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-09T00:20:30.045 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-09T00:20:30.047 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.086 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.052) 0:03:22.059 ********** 2020-03-09T00:20:30.121 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.033) 0:03:22.093 ********** 2020-03-09T00:20:30.127 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-09T00:20:30.129 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.167 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.046) 0:03:22.139 ********** 2020-03-09T00:20:30.202 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.035) 0:03:22.175 ********** 2020-03-09T00:20:30.209 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-09T00:20:30.209 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.248 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.045) 0:03:22.221 ********** 2020-03-09T00:20:30.255 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.294 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.045) 0:03:22.266 ********** 2020-03-09T00:20:30.299 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-09T00:20:30.311 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.314 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-09T00:20:30.316 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:13:35 UTC", "ActiveEnterTimestampMonotonic": "14890338", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target tmp.mount ntpdate.service ntpd.service sysinit.target sntp.service systemd-journald.socket system.slice -.mount systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:13:34 UTC", "AssertTimestampMonotonic": "13432515", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:13:34 UTC", "ConditionTimestampMonotonic": "13432471", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-09T00:20:30.318 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2348", "ExecMainStartTimestamp": "Mon 2020-03-09 00:13:35 UTC", "ExecMainStartTimestampMonotonic": "14754557", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:13:34 UTC", "InactiveExitTimestampMonotonic": "13451018", "InvocationID": "7dea7ab3a50d433d809fc1c44407e6d1", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-09T00:20:30.319 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127443", "LimitNPROCSoft": "127443", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127443", "LimitSIGPENDINGSoft": "127443", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2348", "MemoryAccounting": "yes", "MemoryCurrent": "2662400", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-09T00:20:30.321 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:13:35 UTC", "StateChangeTimestampMonotonic": "14890338", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:13:35 UTC", "WatchdogTimestampMonotonic": "14754566", "WatchdogUSec": "0"}} 2020-03-09T00:20:30.360 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.066) 0:03:22.332 ********** 2020-03-09T00:20:30.399 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.038) 0:03:22.371 ********** 2020-03-09T00:20:30.406 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-09T00:20:30.407 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-09T00:20:30.408 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.446 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.047) 0:03:22.419 ********** 2020-03-09T00:20:30.481 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.034) 0:03:22.454 ********** 2020-03-09T00:20:30.517 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.035) 0:03:22.490 ********** 2020-03-09T00:20:30.523 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-09T00:20:30.523 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.525 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-09T00:20:30.525 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-09T00:20:30.562 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.045) 0:03:22.535 ********** 2020-03-09T00:20:30.596 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.033) 0:03:22.568 ********** 2020-03-09T00:20:30.630 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.034) 0:03:22.602 ********** 2020-03-09T00:20:30.635 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-09T00:20:30.637 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.676 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.046) 0:03:22.649 ********** 2020-03-09T00:20:30.711 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.034) 0:03:22.683 ********** 2020-03-09T00:20:30.717 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-09T00:20:30.719 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.759 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.047) 0:03:22.731 ********** 2020-03-09T00:20:30.766 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:30.803 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.044) 0:03:22.775 ********** 2020-03-09T00:20:30.932 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-09T00:20:30.934 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true} 2020-03-09T00:20:30.970 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:30 +0000 (0:00:00.167) 0:03:22.943 ********** 2020-03-09T00:20:30.976 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-09T00:20:30.977 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-09T00:20:31.016 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.045) 0:03:22.988 ********** 2020-03-09T00:20:31.076 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-09T00:20:31.078 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-09T00:20:31.114 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.098) 0:03:23.087 ********** 2020-03-09T00:20:31.282 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-09T00:20:31.319 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.204) 0:03:23.292 ********** 2020-03-09T00:20:31.324 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-09T00:20:31.325 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-09T00:20:31.364 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.044) 0:03:23.337 ********** 2020-03-09T00:20:31.401 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-09T00:20:31.403 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.441 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.076) 0:03:23.414 ********** 2020-03-09T00:20:31.447 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-09T00:20:31.449 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:13:06 UTC", "ActiveEnterTimestampMonotonic": "17242434", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service tmp.mount system.slice sntp.service basic.target -.mount systemd-tmpfiles-setup.service ntpd.service systemd-journald.socket sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:13:06 UTC", "AssertTimestampMonotonic": "16452437", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:13:06 UTC", "ConditionTimestampMonotonic": "16452410", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-09T00:20:31.451 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2328", "ExecMainStartTimestamp": "Mon 2020-03-09 00:13:06 UTC", "ExecMainStartTimestampMonotonic": "16668402", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:13:06 UTC", "InactiveExitTimestampMonotonic": "16461265", "InvocationID": "f6ecb35fa520484293a3da3f0b64f1b3", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-09T00:20:31.453 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127467", "LimitNPROCSoft": "127467", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127467", "LimitSIGPENDINGSoft": "127467", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2328", "MemoryAccounting": "yes", "MemoryCurrent": "3092480", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-09T00:20:31.454 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:13:06 UTC", "StateChangeTimestampMonotonic": "17242434", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:13:06 UTC", "WatchdogTimestampMonotonic": "16668416", "WatchdogUSec": "0"}} 2020-03-09T00:20:31.489 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.047) 0:03:23.461 ********** 2020-03-09T00:20:31.494 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-09T00:20:31.495 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.532 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.043) 0:03:23.505 ********** 2020-03-09T00:20:31.537 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-09T00:20:31.538 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-09T00:20:31.540 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.575 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.043) 0:03:23.548 ********** 2020-03-09T00:20:31.608 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.033) 0:03:23.581 ********** 2020-03-09T00:20:31.615 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-09T00:20:31.615 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.617 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.654 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.045) 0:03:23.627 ********** 2020-03-09T00:20:31.689 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.034) 0:03:23.661 ********** 2020-03-09T00:20:31.695 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-09T00:20:31.697 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:16:36 UTC", "ActiveEnterTimestampMonotonic": "10981455", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target sntp.service tmp.mount system.slice ntpd.service ntpdate.service systemd-journald.socket -.mount systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:16:35 UTC", "AssertTimestampMonotonic": "9501096", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:16:35 UTC", "ConditionTimestampMonotonic": "9501070", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": 2020-03-09T00:20:31.699 INFO:teuthology.task.ansible.out:"NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1486", "ExecMainStartTimestamp": "Mon 2020-03-09 00:16:35 UTC", "ExecMainStartTimestampMonotonic": "9669070", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:16:35 UTC", "InactiveExitTimestampMonotonic": "9550139", "InvocationID": "427a00542eec4802b715ae76ab1887f4", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", 2020-03-09T00:20:31.700 INFO:teuthology.task.ansible.out: "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127437", "LimitNPROCSoft": "127437", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127437", "LimitSIGPENDINGSoft": "127437", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1486", "MemoryAccounting": "yes", "MemoryCurrent": "3551232", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError 2020-03-09T00:20:31.702 INFO:teuthology.task.ansible.out:": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:16:36 UTC", "StateChangeTimestampMonotonic": "10981455", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:16:35 UTC", "WatchdogTimestampMonotonic": "9669080", "WatchdogUSec": "0"}} TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.736 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.047) 0:03:23.709 ********** 2020-03-09T00:20:31.770 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.033) 0:03:23.742 ********** 2020-03-09T00:20:31.775 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-09T00:20:31.776 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.816 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.045) 0:03:23.788 ********** 2020-03-09T00:20:31.822 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-09T00:20:31.822 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.861 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.044) 0:03:23.833 ********** 2020-03-09T00:20:31.867 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.906 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.045) 0:03:23.878 ********** 2020-03-09T00:20:31.911 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-09T00:20:31.912 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.951 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.044) 0:03:23.923 ********** 2020-03-09T00:20:31.956 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:31.957 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-09T00:20:31.957 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-09T00:20:32.001 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:31 +0000 (0:00:00.050) 0:03:23.974 ********** 2020-03-09T00:20:32.037 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.035) 0:03:24.009 ********** 2020-03-09T00:20:32.043 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-09T00:20:32.043 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:32.083 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.046) 0:03:24.056 ********** 2020-03-09T00:20:32.089 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-09T00:20:32.090 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:32.129 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.045) 0:03:24.101 ********** 2020-03-09T00:20:32.170 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:32.196 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.066) 0:03:24.167 ********** 2020-03-09T00:20:32.222 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-09T00:20:32.224 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:32.259 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.064) 0:03:24.232 ********** 2020-03-09T00:20:32.286 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-09T00:20:32.288 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:32.328 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.064) 0:03:24.296 ********** 2020-03-09T00:20:32.330 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-09T00:20:32.331 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-09T00:20:32.370 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.046) 0:03:24.343 ********** 2020-03-09T00:20:32.375 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-09T00:20:32.375 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:32.415 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.045) 0:03:24.388 ********** 2020-03-09T00:20:32.548 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-09T00:20:32.549 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-09T00:20:32.586 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.170) 0:03:24.558 ********** 2020-03-09T00:20:32.868 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-09T00:20:32.910 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.323) 0:03:24.882 ********** 2020-03-09T00:20:32.917 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-09T00:20:32.918 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-09T00:20:32.962 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:32 +0000 (0:00:00.052) 0:03:24.935 ********** 2020-03-09T00:20:33.001 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-09T00:20:33.002 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-09T00:20:33.060 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.098) 0:03:25.033 ********** 2020-03-09T00:20:33.087 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-09T00:20:33.089 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:33.128 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.067) 0:03:25.100 ********** 2020-03-09T00:20:33.155 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-09T00:20:33.157 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-09T00:20:33.194 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.065) 0:03:25.166 ********** 2020-03-09T00:20:33.274 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:33.276 INFO:teuthology.task.ansible.out:TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-09T00:20:33.278 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-03-09 00:16:39 UTC", "ActiveEnterTimestampMonotonic": "11916143", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target ntpdate.service systemd-journald.socket sntp.service systemd-tmpfiles-setup.service ntpd.service system.slice tmp.mount -.mount sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-03-09 00:16:38 UTC", "AssertTimestampMonotonic": "10597668", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-03-09 00:16:38 UTC", "ConditionTimestampMonotonic": "10597641", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-09T00:20:33.280 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2315", "ExecMainStartTimestamp": "Mon 2020-03-09 00:16:39 UTC", "ExecMainStartTimestampMonotonic": "11869909", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-03-09 00:16:38 UTC", "InactiveExitTimestampMonotonic": "10611115", "InvocationID": "59d234c203314e1e93c7540b056b4fde", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-09T00:20:33.281 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2315", "MemoryAccounting": "yes", "MemoryCurrent": "3141632", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-09T00:20:33.283 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-03-09 00:16:39 UTC", "StateChangeTimestampMonotonic": "11916143", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-03-09 00:16:39 UTC", "WatchdogTimestampMonotonic": "11869923", "WatchdogUSec": "0"}} 2020-03-09T00:20:33.317 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.123) 0:03:25.290 ********** 2020-03-09T00:20:33.345 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-09T00:20:33.346 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:33.383 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.065) 0:03:25.355 ********** 2020-03-09T00:20:33.399 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-09T00:20:33.401 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-09T00:20:33.442 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.058) 0:03:25.414 ********** 2020-03-09T00:20:33.448 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-09T00:20:33.450 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:33.487 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.045) 0:03:25.460 ********** 2020-03-09T00:20:33.494 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-09T00:20:33.495 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:33.534 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.046) 0:03:25.506 ********** 2020-03-09T00:20:33.540 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-09T00:20:33.541 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:33.579 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.044) 0:03:25.551 ********** 2020-03-09T00:20:33.584 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-09T00:20:33.585 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-09T00:20:33.627 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.048) 0:03:25.600 ********** 2020-03-09T00:20:33.654 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-09T00:20:33.657 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:33.693 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.065) 0:03:25.666 ********** 2020-03-09T00:20:33.754 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-09T00:20:33.755 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-09T00:20:33.799 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.103) 0:03:25.770 ********** 2020-03-09T00:20:33.804 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-09T00:20:33.805 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-09T00:20:33.844 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.047) 0:03:25.817 ********** 2020-03-09T00:20:33.851 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-09T00:20:33.851 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:33.893 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.049) 0:03:25.866 ********** 2020-03-09T00:20:33.898 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-09T00:20:33.899 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-09T00:20:33.938 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.044) 0:03:25.911 ********** 2020-03-09T00:20:33.946 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-09T00:20:33.948 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-03-09T00:20:33.985 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:33 +0000 (0:00:00.046) 0:03:25.958 ********** 2020-03-09T00:20:34.055 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-09T00:20:34.056 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-09T00:20:34.092 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.107) 0:03:26.065 ********** 2020-03-09T00:20:34.132 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-09T00:20:34.134 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.168 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.075) 0:03:26.140 ********** 2020-03-09T00:20:34.173 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-09T00:20:34.174 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-09T00:20:34.213 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.045) 0:03:26.185 ********** 2020-03-09T00:20:34.218 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-09T00:20:34.219 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-09T00:20:34.221 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-09T00:20:34.258 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.045) 0:03:26.231 ********** 2020-03-09T00:20:34.292 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.033) 0:03:26.265 ********** 2020-03-09T00:20:34.297 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-09T00:20:34.299 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.339 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.046) 0:03:26.311 ********** 2020-03-09T00:20:34.374 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.034) 0:03:26.346 ********** 2020-03-09T00:20:34.378 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-09T00:20:34.379 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-09T00:20:34.381 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-09T00:20:34.424 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.050) 0:03:26.397 ********** 2020-03-09T00:20:34.474 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.037) 0:03:26.435 ********** TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi104.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.512 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.050) 0:03:26.485 ********** 2020-03-09T00:20:34.548 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.035) 0:03:26.521 ********** 2020-03-09T00:20:34.556 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-09T00:20:34.557 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-09T00:20:34.600 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.052) 0:03:26.573 ********** 2020-03-09T00:20:34.607 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-09T00:20:34.608 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.610 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.612 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi104.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.624 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.626 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-09T00:20:34.626 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.668 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.067) 0:03:26.640 ********** 2020-03-09T00:20:34.710 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.036) 0:03:26.676 ********** TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:34.712 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.753 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.048) 0:03:26.725 ********** 2020-03-09T00:20:34.759 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-09T00:20:34.761 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-09T00:20:34.761 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-09T00:20:34.800 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.047) 0:03:26.773 ********** 2020-03-09T00:20:34.837 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.036) 0:03:26.810 ********** 2020-03-09T00:20:34.843 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-09T00:20:34.845 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.883 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.045) 0:03:26.856 ********** 2020-03-09T00:20:34.891 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-09T00:20:34.892 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.930 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.046) 0:03:26.903 ********** 2020-03-09T00:20:34.936 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-09T00:20:34.938 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:34.980 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:34 +0000 (0:00:00.049) 0:03:26.952 ********** 2020-03-09T00:20:35.008 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-09T00:20:35.009 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-09T00:20:35.046 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.066) 0:03:27.019 ********** 2020-03-09T00:20:35.082 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.035) 0:03:27.055 ********** 2020-03-09T00:20:35.089 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-09T00:20:35.090 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.130 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.047) 0:03:27.103 ********** 2020-03-09T00:20:35.136 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-09T00:20:35.138 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-09T00:20:35.181 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.051) 0:03:27.154 ********** 2020-03-09T00:20:35.217 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.035) 0:03:27.190 ********** 2020-03-09T00:20:35.223 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-09T00:20:35.224 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.226 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.227 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.229 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.268 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.050) 0:03:27.240 ********** 2020-03-09T00:20:35.309 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.040) 0:03:27.281 ********** 2020-03-09T00:20:35.315 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-09T00:20:35.316 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.317 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.328 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.340 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-09T00:20:35.341 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.341 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi050.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.343 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.384 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.074) 0:03:27.356 ********** 2020-03-09T00:20:35.437 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.052) 0:03:27.409 ********** 2020-03-09T00:20:35.444 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:35.445 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.446 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi148.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-09T00:20:35.499 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.062) 0:03:27.472 ********** 2020-03-09T00:20:35.537 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.037) 0:03:27.509 ********** 2020-03-09T00:20:35.544 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-09T00:20:35.545 INFO:teuthology.task.ansible.out:skipping: [smithi104.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.546 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.585 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.048) 0:03:27.558 ********** 2020-03-09T00:20:35.625 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.039) 0:03:27.597 ********** 2020-03-09T00:20:35.631 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-09T00:20:35.631 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.633 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-09T00:20:35.635 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.678 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.053) 0:03:27.650 ********** 2020-03-09T00:20:35.723 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.045) 0:03:27.696 ********** 2020-03-09T00:20:35.770 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.047) 0:03:27.743 ********** 2020-03-09T00:20:35.776 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-09T00:20:35.778 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Purge resolvconf] ********************************************* 2020-03-09T00:20:35.779 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.850 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.079) 0:03:27.823 ********** 2020-03-09T00:20:35.887 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.036) 0:03:27.859 ********** 2020-03-09T00:20:35.893 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-09T00:20:35.894 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.895 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.896 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.897 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.899 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:35.957 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.070) 0:03:27.930 ********** 2020-03-09T00:20:35.994 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:35 +0000 (0:00:00.036) 0:03:27.966 ********** 2020-03-09T00:20:35.999 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-09T00:20:36.000 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-09T00:20:36.002 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.079 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.085) 0:03:28.051 ********** 2020-03-09T00:20:36.118 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.039) 0:03:28.091 ********** 2020-03-09T00:20:36.124 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-09T00:20:36.124 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.126 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.126 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.128 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi148.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.130 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-09T00:20:36.132 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.172 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.054) 0:03:28.145 ********** 2020-03-09T00:20:36.207 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.034) 0:03:28.179 ********** 2020-03-09T00:20:36.212 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-09T00:20:36.213 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.250 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.043) 0:03:28.223 ********** 2020-03-09T00:20:36.256 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:36.258 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.298 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.047) 0:03:28.270 ********** 2020-03-09T00:20:36.314 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-09T00:20:36.316 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-09T00:20:36.352 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.054) 0:03:28.325 ********** 2020-03-09T00:20:36.359 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-09T00:20:36.361 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.399 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.046) 0:03:28.371 ********** 2020-03-09T00:20:36.404 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-09T00:20:36.406 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.447 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.048) 0:03:28.420 ********** 2020-03-09T00:20:36.453 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-09T00:20:36.455 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.497 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.049) 0:03:28.469 ********** 2020-03-09T00:20:36.502 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:36.504 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005649", "end": "2020-03-09 00:20:36.456006", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-09 00:20:36.450357", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:36.542 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.045) 0:03:28.515 ********** 2020-03-09T00:20:36.548 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-09T00:20:36.550 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.587 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.044) 0:03:28.560 ********** 2020-03-09T00:20:36.593 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-09T00:20:36.594 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.631 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.043) 0:03:28.603 ********** 2020-03-09T00:20:36.635 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-09T00:20:36.638 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.639 INFO:teuthology.task.ansible.out: 2020-03-09T00:20:36.680 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.049) 0:03:28.653 ********** 2020-03-09T00:20:36.687 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:36.688 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003197", "end": "2020-03-09 00:20:36.654049", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-09 00:20:36.650852", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:36.710 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-09T00:20:36.712 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.754 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.073) 0:03:28.726 ********** 2020-03-09T00:20:36.780 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-09T00:20:36.782 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.821 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.067) 0:03:28.794 ********** 2020-03-09T00:20:36.840 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-09T00:20:36.842 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-09T00:20:36.879 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.057) 0:03:28.852 ********** 2020-03-09T00:20:36.885 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-09T00:20:36.886 INFO:teuthology.task.ansible.out:skipping: [smithi148.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:36.888 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi104.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006169", "end": "2020-03-09 00:20:36.813761", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-09 00:20:35.807592", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:36.889 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003048", "end": "2020-03-09 00:20:36.851944", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:20:36.848896", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:36.943 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:36 +0000 (0:00:00.063) 0:03:28.915 ********** 2020-03-09T00:20:37.046 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003027", "end": "2020-03-09 00:20:37.013827", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-09 00:20:37.010800", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:37.079 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-09T00:20:37.081 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-09T00:20:37.120 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.177) 0:03:29.093 ********** 2020-03-09T00:20:37.126 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:37.127 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.020733", "end": "2020-03-09 00:20:37.075872", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-09 00:20:37.055139", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:37.170 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.049) 0:03:29.142 ********** 2020-03-09T00:20:37.176 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-09T00:20:37.178 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:37.180 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004951", "end": "2020-03-09 00:20:37.135358", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-09 00:20:37.130407", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:37.217 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.047) 0:03:29.190 ********** 2020-03-09T00:20:37.222 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-09T00:20:37.224 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:37.225 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002715", "end": "2020-03-09 00:20:37.195886", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:20:37.193171", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:37.267 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.050) 0:03:29.240 ********** 2020-03-09T00:20:37.275 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-09T00:20:37.277 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.277 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.278 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.320 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.052) 0:03:29.292 ********** 2020-03-09T00:20:37.361 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:37.362 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002998", "end": "2020-03-09 00:20:37.321712", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-09 00:20:37.318714", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:37.362 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-09T00:20:37.362 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.364 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.375 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.386 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.388 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.430 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.110) 0:03:29.403 ********** 2020-03-09T00:20:37.436 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:37.437 INFO:teuthology.task.ansible.out:ok: [smithi104.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.021452", "end": "2020-03-09 00:20:37.394232", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-09 00:20:37.372780", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:37.481 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.050) 0:03:29.453 ********** 2020-03-09T00:20:37.486 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-09T00:20:37.488 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:20:37.556 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.074) 0:03:29.528 ********** 2020-03-09T00:20:37.562 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-09T00:20:37.625 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.069) 0:03:29.597 ********** 2020-03-09T00:20:37.631 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-09T00:20:37.633 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.672 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.046) 0:03:29.644 ********** 2020-03-09T00:20:37.698 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-09T00:20:37.700 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.738 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.066) 0:03:29.711 ********** 2020-03-09T00:20:37.765 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.806 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.064) 0:03:29.775 ********** 2020-03-09T00:20:37.819 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-09T00:20:37.822 INFO:teuthology.task.ansible.out:changed: [smithi104.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:20:37.861 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-09T00:20:37.863 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.921 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.118) 0:03:29.894 ********** 2020-03-09T00:20:37.952 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-09T00:20:37.954 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:37.992 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:37 +0000 (0:00:00.070) 0:03:29.964 ********** 2020-03-09T00:20:38.019 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-09T00:20:38.021 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:38.059 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:38 +0000 (0:00:00.066) 0:03:30.031 ********** 2020-03-09T00:20:38.084 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-09T00:20:38.086 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-09T00:20:38.120 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:38 +0000 (0:00:00.061) 0:03:30.093 ********** 2020-03-09T00:20:38.355 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:38.357 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005016", "end": "2020-03-09 00:20:38.311110", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-09 00:20:38.306094", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:38.532 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003108", "end": "2020-03-09 00:20:38.501514", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-09 00:20:38.498406", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:38.554 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:01.003885", "end": "2020-03-09 00:20:38.509154", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:20:37.505269", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:38.721 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002799", "end": "2020-03-09 00:20:38.686968", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-09 00:20:38.684169", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:38.763 INFO:teuthology.task.ansible.out:ok: [smithi148.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015213", "end": "2020-03-09 00:20:38.727793", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-09 00:20:38.712580", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:38.805 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:38 +0000 (0:00:00.684) 0:03:30.777 ********** 2020-03-09T00:20:38.832 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-09T00:20:38.869 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:38 +0000 (0:00:00.064) 0:03:30.842 ********** 2020-03-09T00:20:38.919 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-09T00:20:38.921 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.025738", "end": "2020-03-09 00:20:38.889347", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-09 00:20:38.863609", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-09T00:20:38.962 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:38 +0000 (0:00:00.092) 0:03:30.935 ********** 2020-03-09T00:20:38.989 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-09T00:20:39.027 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:39 +0000 (0:00:00.065) 0:03:31.000 ********** 2020-03-09T00:20:39.065 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-09T00:20:39.066 INFO:teuthology.task.ansible.out:changed: [smithi148.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:20:39.234 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-09T00:20:39.332 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-03-09T00:20:39.334 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-03-09T00:20:39.335 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-03-09T00:20:39.336 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi050.front.sepia.ceph.com : ok=115 changed=36 unreachable=0 failed=0 skipped=166 rescued=0 ignored=3 smithi104.front.sepia.ceph.com : ok=115 changed=37 unreachable=0 failed=0 skipped=166 rescued=0 ignored=3 2020-03-09T00:20:39.336 INFO:teuthology.task.ansible.out:smithi139.front.sepia.ceph.com : ok=115 changed=37 unreachable=0 failed=0 skipped=166 rescued=0 ignored=3 smithi148.front.sepia.ceph.com : ok=115 changed=37 unreachable=0 failed=0 skipped=166 rescued=0 ignored=3 2020-03-09T00:20:39.336 INFO:teuthology.task.ansible.out:Monday 09 March 2020 00:20:39 +0000 (0:00:00.308) 0:03:31.309 ********** =============================================================================== 2020-03-09T00:20:39.337 INFO:teuthology.task.ansible.out:users : Create all admin users with sudo access. ----------------------- 23.82s users : Update authorized_keys using the keys repo --------------------- 23.67s common : nrpe - Remove SELinux policy package -------------------------- 14.25s common : nrpe - Load SELinux policy package ---------------------------- 13.97s testnode : Base group workaround for BZ1782899 ------------------------- 10.83s testnode : Install packages --------------------------------------------- 9.45s users : Remove revoked users -------------------------------------------- 6.33s testnode : Zap all non-root disks --------------------------------------- 4.42s Gathering Facts --------------------------------------------------------- 2.63s testnode : Ensure ceph packages are not present. ------------------------ 1.72s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.07s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.87s 2020-03-09T00:20:39.338 INFO:teuthology.task.ansible.out:testnode : Ensure ceph dependency packages are not present. ------------- 0.87s ansible-managed : Add authorized keys for the ansible user. ------------- 0.85s users : Clone the keys repo --------------------------------------------- 0.74s Gathering Facts --------------------------------------------------------- 0.72s testnode : List any leftover Ceph artifacts from previous jobs ---------- 0.68s common : Make sure smartmontools is installed --------------------------- 0.51s users : Check keys_repo HEAD sha1 --------------------------------------- 0.51s ansible-managed : Create the ansible user. ------------------------------ 0.50s 2020-03-09T00:20:39.650 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi148.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com')] 2020-03-09T00:20:39.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:20:39.653 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi148.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:20:39.730 INFO:teuthology.orchestra.run.smithi148:> true 2020-03-09T00:20:39.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:20:39.820 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi104.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:20:39.895 INFO:teuthology.orchestra.run.smithi104:> true 2020-03-09T00:20:39.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:20:39.993 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi050.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:20:40.076 INFO:teuthology.orchestra.run.smithi050:> true 2020-03-09T00:20:40.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-09T00:20:40.169 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi139.front.sepia.ceph.com', 'timeout': 60} 2020-03-09T00:20:40.243 INFO:teuthology.orchestra.run.smithi139:> true 2020-03-09T00:20:40.335 INFO:teuthology.run_tasks:Running task clock... 2020-03-09T00:20:40.348 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-09T00:20:40.349 INFO:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-09T00:20:40.389 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-09T00:20:40.408 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-09T00:20:40.450 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2020-03-09T00:20:40.465 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2020-03-09T00:20:40.482 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-09T00:20:40.500 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-09T00:20:40.561 INFO:teuthology.orchestra.run.smithi148.stderr:bash: ntpq: command not found 2020-03-09T00:20:40.569 INFO:teuthology.orchestra.run.smithi148.stdout:210 Number of sources = 4 2020-03-09T00:20:40.569 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-09T00:20:40.570 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2020-03-09T00:20:40.570 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.570 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.570 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.570 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.571 INFO:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-09T00:20:40.610 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-09T00:20:40.630 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-09T00:20:40.681 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2020-03-09T00:20:40.694 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2020-03-09T00:20:40.712 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-09T00:20:40.729 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-09T00:20:40.795 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2020-03-09T00:20:40.797 INFO:teuthology.orchestra.run.smithi104.stdout:210 Number of sources = 4 2020-03-09T00:20:40.798 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-09T00:20:40.798 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2020-03-09T00:20:40.798 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.798 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.798 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.798 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:40.799 INFO:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-09T00:20:40.844 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-09T00:20:40.867 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-09T00:20:40.918 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2020-03-09T00:20:40.938 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2020-03-09T00:20:40.958 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-09T00:20:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-09T00:20:41.044 INFO:teuthology.orchestra.run.smithi050.stderr:bash: ntpq: command not found 2020-03-09T00:20:41.047 INFO:teuthology.orchestra.run.smithi050.stdout:210 Number of sources = 4 2020-03-09T00:20:41.047 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-09T00:20:41.047 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2020-03-09T00:20:41.048 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.048 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.048 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.048 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.049 INFO:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-09T00:20:41.088 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-09T00:20:41.107 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-09T00:20:41.150 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2020-03-09T00:20:41.165 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2020-03-09T00:20:41.182 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-09T00:20:41.200 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-09T00:20:41.265 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2020-03-09T00:20:41.268 INFO:teuthology.orchestra.run.smithi139.stdout:210 Number of sources = 4 2020-03-09T00:20:41.268 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-09T00:20:41.268 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2020-03-09T00:20:41.269 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.269 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.269 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.269 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-09T00:20:41.269 INFO:teuthology.run_tasks:Running task install... 2020-03-09T00:20:41.278 DEBUG:teuthology.task.install:project ceph 2020-03-09T00:20:41.278 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b'}} 2020-03-09T00:20:41.278 DEBUG:teuthology.task.install:config {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b'} 2020-03-09T00:20:41.279 INFO:teuthology.task.install:Using flavor: basic 2020-03-09T00:20:41.287 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-09T00:20:41.287 INFO:teuthology.task.install:extra packages: [] 2020-03-09T00:20:41.288 INFO:teuthology.orchestra.run.smithi148:> sudo lsb_release -is 2020-03-09T00:20:41.327 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS 2020-03-09T00:20:41.328 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.328 INFO:teuthology.orchestra.run.smithi104:> sudo lsb_release -is 2020-03-09T00:20:41.328 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-09T00:20:41.329 INFO:teuthology.orchestra.run.smithi148:> sudo lsb_release -is 2020-03-09T00:20:41.366 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS 2020-03-09T00:20:41.368 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.368 INFO:teuthology.orchestra.run.smithi050:> sudo lsb_release -is 2020-03-09T00:20:41.368 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-09T00:20:41.368 INFO:teuthology.orchestra.run.smithi104:> sudo lsb_release -is 2020-03-09T00:20:41.406 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS 2020-03-09T00:20:41.407 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.407 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:20:41.412 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS 2020-03-09T00:20:41.413 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.413 INFO:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2020-03-09T00:20:41.413 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-09T00:20:41.414 INFO:teuthology.orchestra.run.smithi050:> sudo lsb_release -is 2020-03-09T00:20:41.446 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS 2020-03-09T00:20:41.448 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:20:41.454 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS 2020-03-09T00:20:41.454 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.455 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-09T00:20:41.455 INFO:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2020-03-09T00:20:41.496 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS 2020-03-09T00:20:41.497 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.498 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:20:41.534 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS 2020-03-09T00:20:41.534 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:20:41.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:20:41.599 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/ 2020-03-09T00:20:41.600 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1904.g9eb9490 2020-03-09T00:20:41.618 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/ 2020-03-09T00:20:41.618 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1904.g9eb9490 2020-03-09T00:20:41.654 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/ 2020-03-09T00:20:41.654 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1904.g9eb9490 2020-03-09T00:20:41.711 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/ 2020-03-09T00:20:41.711 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1904.g9eb9490 2020-03-09T00:20:41.732 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-09T00:20:41.732 INFO:teuthology.orchestra.run.smithi148:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-09T00:20:41.743 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-09T00:20:41.743 INFO:teuthology.orchestra.run.smithi104:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-09T00:20:41.782 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-09T00:20:41.782 INFO:teuthology.orchestra.run.smithi050:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-09T00:20:41.787 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-09T00:20:41.788 INFO:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9eb9490ed0531fd33292066429bf9db55b9c1d3b/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-09T00:20:41.790 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-09T00:20:41.790 INFO:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9eb9490ed0531fd33292066429bf9db55b9c1d3b/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-09T00:20:41.822 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage4-testing-2020-03-08-0938/9eb9490ed0531fd33292066429bf9db55b9c1d3b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-09T00:20:41.822 INFO:teuthology.orchestra.run.smithi139:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-09T00:20:41.828 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-09T00:20:41.829 INFO:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9eb9490ed0531fd33292066429bf9db55b9c1d3b/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-09T00:20:41.858 INFO:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-09T00:20:41.859 INFO:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-09T00:20:41.867 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-09T00:20:41.867 INFO:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9eb9490ed0531fd33292066429bf9db55b9c1d3b/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-09T00:20:41.901 INFO:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-09T00:20:41.936 INFO:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-09T00:20:41.938 INFO:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-09T00:20:41.942 INFO:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-09T00:20:41.991 INFO:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-09T00:20:42.005 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2020-03-09T00:20:42.006 INFO:teuthology.orchestra.run.smithi104:> sudo yum clean all 2020-03-09T00:20:42.009 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2020-03-09T00:20:42.011 INFO:teuthology.orchestra.run.smithi148:> sudo yum clean all 2020-03-09T00:20:42.019 INFO:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-09T00:20:42.022 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2020-03-09T00:20:42.023 INFO:teuthology.orchestra.run.smithi050:> sudo yum clean all 2020-03-09T00:20:42.085 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2020-03-09T00:20:42.086 INFO:teuthology.orchestra.run.smithi139:> sudo yum clean all 2020-03-09T00:20:42.390 INFO:teuthology.orchestra.run.smithi148.stdout:239 files removed 2020-03-09T00:20:42.391 INFO:teuthology.orchestra.run.smithi104.stdout:239 files removed 2020-03-09T00:20:42.411 INFO:teuthology.orchestra.run.smithi050.stdout:239 files removed 2020-03-09T00:20:42.419 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw 2020-03-09T00:20:42.421 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2020-03-09T00:20:42.440 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2020-03-09T00:20:42.469 INFO:teuthology.orchestra.run.smithi139.stdout:239 files removed 2020-03-09T00:20:42.498 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw 2020-03-09T00:20:42.986 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 308 kB/s | 69 kB 00:00 2020-03-09T00:20:42.986 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 309 kB/s | 69 kB 00:00 2020-03-09T00:20:43.029 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 310 kB/s | 69 kB 00:00 2020-03-09T00:20:43.064 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 306 kB/s | 69 kB 00:00 2020-03-09T00:20:43.213 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 68 kB/s | 14 kB 00:00 2020-03-09T00:20:43.215 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 67 kB/s | 14 kB 00:00 2020-03-09T00:20:43.303 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 64 kB/s | 14 kB 00:00 2020-03-09T00:20:43.418 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 7.5 kB/s | 1.4 kB 00:00 2020-03-09T00:20:43.419 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 7.6 kB/s | 1.4 kB 00:00 2020-03-09T00:20:43.507 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 7.6 kB/s | 1.4 kB 00:00 2020-03-09T00:20:43.508 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 30 kB/s | 14 kB 00:00 2020-03-09T00:20:43.714 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 7.6 kB/s | 1.4 kB 00:00 2020-03-09T00:20:44.136 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - AppStream 9.3 MB/s | 6.5 MB 00:00 2020-03-09T00:20:44.186 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - AppStream 9.9 MB/s | 6.5 MB 00:00 2020-03-09T00:20:44.235 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - AppStream 8.2 MB/s | 6.5 MB 00:00 2020-03-09T00:20:44.324 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - AppStream 11 MB/s | 6.5 MB 00:00 2020-03-09T00:20:45.947 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - Base 6.4 MB/s | 5.0 MB 00:00 2020-03-09T00:20:45.960 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Base 6.7 MB/s | 5.0 MB 00:00 2020-03-09T00:20:46.170 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - Base 5.6 MB/s | 5.0 MB 00:00 2020-03-09T00:20:46.204 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - Base 6.4 MB/s | 5.0 MB 00:00 2020-03-09T00:20:47.235 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Extras 3.6 kB/s | 2.1 kB 00:00 2020-03-09T00:20:47.377 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-8 - Extras 2.9 kB/s | 2.1 kB 00:00 2020-03-09T00:20:47.565 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - Extras 3.1 kB/s | 2.1 kB 00:00 2020-03-09T00:20:47.617 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-8 - Extras 3.2 kB/s | 2.1 kB 00:00 2020-03-09T00:20:47.672 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for ceph-el8 owned by ktdreyer 152 kB/s | 38 kB 00:00 2020-03-09T00:20:47.673 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for ceph-el8 owned by ktdreyer 97 kB/s | 38 kB 00:00 2020-03-09T00:20:47.724 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for ceph-el8 owned by ktdreyer 337 kB/s | 38 kB 00:00 2020-03-09T00:20:47.772 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for ceph-el8 owned by ktdreyer 358 kB/s | 38 kB 00:00 2020-03-09T00:20:48.079 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 17 MB/s | 6.0 MB 00:00 2020-03-09T00:20:48.081 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 17 MB/s | 6.0 MB 00:00 2020-03-09T00:20:48.143 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 16 MB/s | 6.0 MB 00:00 2020-03-09T00:20:48.421 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 10 MB/s | 6.0 MB 00:00 2020-03-09T00:20:49.337 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 5.6 MB/s | 30 kB 00:00 2020-03-09T00:20:49.353 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 5.3 MB/s | 30 kB 00:00 2020-03-09T00:20:49.423 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 5.2 MB/s | 30 kB 00:00 2020-03-09T00:20:49.720 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 5.5 MB/s | 30 kB 00:00 2020-03-09T00:20:50.307 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:20:50.310 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:20:50.310 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2020-03-09T00:20:50.311 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:20:50.311 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:20:50.311 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 9.8 M 2020-03-09T00:20:50.311 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2020-03-09T00:20:50.312 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 3.6 M 2020-03-09T00:20:50.312 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 1.9 M 2020-03-09T00:20:50.312 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:20:50.312 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 5.5 M 2020-03-09T00:20:50.313 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 M 2020-03-09T00:20:50.313 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 25 k 2020-03-09T00:20:50.313 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 507 k 2020-03-09T00:20:50.313 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 377 k 2020-03-09T00:20:50.313 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 6.2 M 2020-03-09T00:20:50.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 40 k 2020-03-09T00:20:50.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 49 k 2020-03-09T00:20:50.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 162 k 2020-03-09T00:20:50.314 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 351 k 2020-03-09T00:20:50.315 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 309 k 2020-03-09T00:20:50.315 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 113 k 2020-03-09T00:20:50.323 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 0.11.4-1.el8 AppStream 353 k 2020-03-09T00:20:50.323 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.8.1-11.el8 AppStream 259 k 2020-03-09T00:20:50.324 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-14.el8 AppStream 44 k 2020-03-09T00:20:50.325 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.9.0-1.el8 BaseOS 47 k 2020-03-09T00:20:50.326 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-09T00:20:50.326 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-09T00:20:50.326 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-09T00:20:50.326 INFO:teuthology.orchestra.run.smithi104.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-09T00:20:50.327 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:20:50.327 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:20:50.327 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:20:50.327 INFO:teuthology.orchestra.run.smithi104.stdout:Install 21 Packages 2020-03-09T00:20:50.327 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2020-03-09T00:20:50.328 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:20:50.330 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:20:50.330 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:20:50.330 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2020-03-09T00:20:50.331 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:20:50.331 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:20:50.331 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 9.8 M 2020-03-09T00:20:50.332 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2020-03-09T00:20:50.332 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 3.6 M 2020-03-09T00:20:50.332 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 1.9 M 2020-03-09T00:20:50.332 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:20:50.334 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 5.5 M 2020-03-09T00:20:50.334 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 M 2020-03-09T00:20:50.335 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 25 k 2020-03-09T00:20:50.335 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 507 k 2020-03-09T00:20:50.336 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 377 k 2020-03-09T00:20:50.336 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 6.2 M 2020-03-09T00:20:50.337 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 40 k 2020-03-09T00:20:50.337 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 49 k 2020-03-09T00:20:50.337 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 162 k 2020-03-09T00:20:50.337 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 351 k 2020-03-09T00:20:50.338 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 309 k 2020-03-09T00:20:50.338 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 113 k 2020-03-09T00:20:50.338 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 0.11.4-1.el8 AppStream 353 k 2020-03-09T00:20:50.338 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.8.1-11.el8 AppStream 259 k 2020-03-09T00:20:50.339 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-14.el8 AppStream 44 k 2020-03-09T00:20:50.339 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.9.0-1.el8 BaseOS 47 k 2020-03-09T00:20:50.339 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-09T00:20:50.339 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-09T00:20:50.339 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-09T00:20:50.340 INFO:teuthology.orchestra.run.smithi139.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-09T00:20:50.340 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:20:50.340 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:20:50.340 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:20:50.341 INFO:teuthology.orchestra.run.smithi139.stdout:Install 21 Packages 2020-03-09T00:20:50.341 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 2 Packages 2020-03-09T00:20:50.341 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:20:50.341 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 51 M 2020-03-09T00:20:50.342 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:20:50.342 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 51 M 2020-03-09T00:20:50.342 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:20:50.392 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:20:50.394 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:20:50.395 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2020-03-09T00:20:50.396 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:20:50.397 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:20:50.397 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 9.8 M 2020-03-09T00:20:50.397 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2020-03-09T00:20:50.397 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 3.6 M 2020-03-09T00:20:50.398 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 1.9 M 2020-03-09T00:20:50.398 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:20:50.398 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 5.5 M 2020-03-09T00:20:50.398 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 M 2020-03-09T00:20:50.399 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 25 k 2020-03-09T00:20:50.399 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 507 k 2020-03-09T00:20:50.399 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 377 k 2020-03-09T00:20:50.399 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 6.2 M 2020-03-09T00:20:50.399 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 40 k 2020-03-09T00:20:50.400 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 49 k 2020-03-09T00:20:50.400 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 162 k 2020-03-09T00:20:50.400 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 351 k 2020-03-09T00:20:50.400 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 309 k 2020-03-09T00:20:50.401 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 113 k 2020-03-09T00:20:50.401 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 0.11.4-1.el8 AppStream 353 k 2020-03-09T00:20:50.401 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.8.1-11.el8 AppStream 259 k 2020-03-09T00:20:50.401 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-14.el8 AppStream 44 k 2020-03-09T00:20:50.402 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.9.0-1.el8 BaseOS 47 k 2020-03-09T00:20:50.402 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-09T00:20:50.402 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-09T00:20:50.402 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-09T00:20:50.403 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-09T00:20:50.403 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:20:50.403 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:20:50.403 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:20:50.404 INFO:teuthology.orchestra.run.smithi148.stdout:Install 21 Packages 2020-03-09T00:20:50.404 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 2 Packages 2020-03-09T00:20:50.404 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:20:50.405 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 51 M 2020-03-09T00:20:50.405 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:20:50.748 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:20:50.751 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:20:50.751 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2020-03-09T00:20:50.752 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:20:50.752 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:20:50.752 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 9.8 M 2020-03-09T00:20:50.752 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2020-03-09T00:20:50.752 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 3.6 M 2020-03-09T00:20:50.752 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 1.9 M 2020-03-09T00:20:50.753 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:20:50.753 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 5.5 M 2020-03-09T00:20:50.753 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 M 2020-03-09T00:20:50.753 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 25 k 2020-03-09T00:20:50.753 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 507 k 2020-03-09T00:20:50.753 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 377 k 2020-03-09T00:20:50.753 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 6.2 M 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 40 k 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 49 k 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 162 k 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 351 k 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 309 k 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 113 k 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 0.11.4-1.el8 AppStream 353 k 2020-03-09T00:20:50.754 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.8.1-11.el8 AppStream 259 k 2020-03-09T00:20:50.755 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-14.el8 AppStream 44 k 2020-03-09T00:20:50.755 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.9.0-1.el8 BaseOS 47 k 2020-03-09T00:20:50.755 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-09T00:20:50.755 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-09T00:20:50.755 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-09T00:20:50.755 INFO:teuthology.orchestra.run.smithi050.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-09T00:20:50.755 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:20:50.756 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:20:50.756 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:20:50.756 INFO:teuthology.orchestra.run.smithi050.stdout:Install 21 Packages 2020-03-09T00:20:50.756 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2020-03-09T00:20:50.756 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:20:50.757 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 51 M 2020-03-09T00:20:50.757 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:20:51.263 INFO:teuthology.orchestra.run.smithi139.stdout:(1/23): ceph-base-15.1.0-1904.g9eb9490.el8.x86_ 8.8 MB/s | 5.5 MB 00:00 2020-03-09T00:20:51.309 INFO:teuthology.orchestra.run.smithi139.stdout:(2/23): ceph-selinux-15.1.0-1904.g9eb9490.el8.x 539 kB/s | 25 kB 00:00 2020-03-09T00:20:51.377 INFO:teuthology.orchestra.run.smithi139.stdout:(3/23): libcephfs2-15.1.0-1904.g9eb9490.el8.x86 7.3 MB/s | 507 kB 00:00 2020-03-09T00:20:51.440 INFO:teuthology.orchestra.run.smithi139.stdout:(4/23): libradosstriper1-15.1.0-1904.g9eb9490.e 5.9 MB/s | 377 kB 00:00 2020-03-09T00:20:52.112 INFO:teuthology.orchestra.run.smithi139.stdout:(5/23): librgw2-15.1.0-1904.g9eb9490.el8.x86_64 9.3 MB/s | 6.2 MB 00:00 2020-03-09T00:20:52.169 INFO:teuthology.orchestra.run.smithi139.stdout:(6/23): python3-ceph-argparse-15.1.0-1904.g9eb9 705 kB/s | 40 kB 00:00 2020-03-09T00:20:52.225 INFO:teuthology.orchestra.run.smithi139.stdout:(7/23): python3-ceph-common-15.1.0-1904.g9eb949 872 kB/s | 49 kB 00:00 2020-03-09T00:20:52.287 INFO:teuthology.orchestra.run.smithi139.stdout:(8/23): python3-cephfs-15.1.0-1904.g9eb9490.el8 2.5 MB/s | 162 kB 00:00 2020-03-09T00:20:52.332 INFO:teuthology.orchestra.run.smithi139.stdout:(9/23): ceph-radosgw-15.1.0-1904.g9eb9490.el8.x 5.8 MB/s | 9.8 MB 00:01 2020-03-09T00:20:52.349 INFO:teuthology.orchestra.run.smithi139.stdout:(10/23): python3-rados-15.1.0-1904.g9eb9490.el8 5.6 MB/s | 351 kB 00:00 2020-03-09T00:20:52.414 INFO:teuthology.orchestra.run.smithi139.stdout:(11/23): python3-rbd-15.1.0-1904.g9eb9490.el8.x 3.7 MB/s | 309 kB 00:00 2020-03-09T00:20:52.418 INFO:teuthology.orchestra.run.smithi139.stdout:(12/23): python3-rgw-15.1.0-1904.g9eb9490.el8.x 1.6 MB/s | 113 kB 00:00 2020-03-09T00:20:52.746 INFO:teuthology.orchestra.run.smithi139.stdout:(13/23): ceph-common-15.1.0-1904.g9eb9490.el8.x 9.8 MB/s | 21 MB 00:02 2020-03-09T00:20:52.747 INFO:teuthology.orchestra.run.smithi139.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 788 kB/s | 259 kB 00:00 2020-03-09T00:20:52.801 INFO:teuthology.orchestra.run.smithi139.stdout:(15/23): librdkafka-0.11.4-1.el8.x86_64.rpm 913 kB/s | 353 kB 00:00 2020-03-09T00:20:52.863 INFO:teuthology.orchestra.run.smithi139.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 376 kB/s | 44 kB 00:00 2020-03-09T00:20:52.865 INFO:teuthology.orchestra.run.smithi148.stdout:(1/23): ceph-base-15.1.0-1904.g9eb9490.el8.x86_ 2.6 MB/s | 5.5 MB 00:02 2020-03-09T00:20:52.887 INFO:teuthology.orchestra.run.smithi148.stdout:(2/23): ceph-selinux-15.1.0-1904.g9eb9490.el8.x 1.1 MB/s | 25 kB 00:00 2020-03-09T00:20:52.891 INFO:teuthology.orchestra.run.smithi139.stdout:(17/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 326 kB/s | 47 kB 00:00 2020-03-09T00:20:52.895 INFO:teuthology.orchestra.run.smithi139.stdout:(18/23): gperftools-libs-2.7-6.el8.x86_64.rpm 3.3 MB/s | 315 kB 00:00 2020-03-09T00:20:52.901 INFO:teuthology.orchestra.run.smithi139.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 33 MB/s | 179 kB 00:00 2020-03-09T00:20:52.938 INFO:teuthology.orchestra.run.smithi139.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 783 kB/s | 59 kB 00:00 2020-03-09T00:20:52.949 INFO:teuthology.orchestra.run.smithi148.stdout:(3/23): libcephfs2-15.1.0-1904.g9eb9490.el8.x86 8.1 MB/s | 507 kB 00:00 2020-03-09T00:20:52.953 INFO:teuthology.orchestra.run.smithi139.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2020-03-09T00:20:53.077 INFO:teuthology.orchestra.run.smithi148.stdout:(4/23): libradosstriper1-15.1.0-1904.g9eb9490.e 2.9 MB/s | 377 kB 00:00 2020-03-09T00:20:53.283 INFO:teuthology.orchestra.run.smithi139.stdout:(22/23): librados2-15.1.0-1904.g9eb9490.el8.x86 9.3 MB/s | 3.6 MB 00:00 2020-03-09T00:20:53.618 INFO:teuthology.orchestra.run.smithi139.stdout:(23/23): librbd1-15.1.0-1904.g9eb9490.el8.x86_6 2.8 MB/s | 1.9 MB 00:00 2020-03-09T00:20:53.620 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:20:53.620 INFO:teuthology.orchestra.run.smithi139.stdout:Total 15 MB/s | 51 MB 00:03 2020-03-09T00:20:53.626 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:20:53.672 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:20:53.673 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:20:53.856 INFO:teuthology.orchestra.run.smithi148.stdout:(5/23): librgw2-15.1.0-1904.g9eb9490.el8.x86_64 8.0 MB/s | 6.2 MB 00:00 2020-03-09T00:20:53.879 INFO:teuthology.orchestra.run.smithi148.stdout:(6/23): python3-ceph-argparse-15.1.0-1904.g9eb9 1.7 MB/s | 40 kB 00:00 2020-03-09T00:20:53.900 INFO:teuthology.orchestra.run.smithi148.stdout:(7/23): python3-ceph-common-15.1.0-1904.g9eb949 2.3 MB/s | 49 kB 00:00 2020-03-09T00:20:53.926 INFO:teuthology.orchestra.run.smithi148.stdout:(8/23): python3-cephfs-15.1.0-1904.g9eb9490.el8 6.2 MB/s | 162 kB 00:00 2020-03-09T00:20:53.975 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:20:53.975 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:20:53.978 INFO:teuthology.orchestra.run.smithi148.stdout:(9/23): python3-rados-15.1.0-1904.g9eb9490.el8. 6.6 MB/s | 351 kB 00:00 2020-03-09T00:20:54.030 INFO:teuthology.orchestra.run.smithi148.stdout:(10/23): python3-rbd-15.1.0-1904.g9eb9490.el8.x 5.9 MB/s | 309 kB 00:00 2020-03-09T00:20:54.055 INFO:teuthology.orchestra.run.smithi148.stdout:(11/23): python3-rgw-15.1.0-1904.g9eb9490.el8.x 4.4 MB/s | 113 kB 00:00 2020-03-09T00:20:54.278 INFO:teuthology.orchestra.run.smithi104.stdout:(1/23): ceph-common-15.1.0-1904.g9eb9490.el8.x8 5.9 MB/s | 21 MB 00:03 2020-03-09T00:20:54.300 INFO:teuthology.orchestra.run.smithi104.stdout:(2/23): ceph-selinux-15.1.0-1904.g9eb9490.el8.x 1.1 MB/s | 25 kB 00:00 2020-03-09T00:20:54.342 INFO:teuthology.orchestra.run.smithi104.stdout:(3/23): libcephfs2-15.1.0-1904.g9eb9490.el8.x86 12 MB/s | 507 kB 00:00 2020-03-09T00:20:54.345 INFO:teuthology.orchestra.run.smithi148.stdout:(12/23): librdkafka-0.11.4-1.el8.x86_64.rpm 1.2 MB/s | 353 kB 00:00 2020-03-09T00:20:54.385 INFO:teuthology.orchestra.run.smithi104.stdout:(4/23): libradosstriper1-15.1.0-1904.g9eb9490.e 8.8 MB/s | 377 kB 00:00 2020-03-09T00:20:54.579 INFO:teuthology.orchestra.run.smithi148.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2020-03-09T00:20:54.698 INFO:teuthology.orchestra.run.smithi148.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 370 kB/s | 44 kB 00:00 2020-03-09T00:20:54.771 INFO:teuthology.orchestra.run.smithi104.stdout:(5/23): ceph-base-15.1.0-1904.g9eb9490.el8.x86_ 1.4 MB/s | 5.5 MB 00:03 2020-03-09T00:20:54.789 INFO:teuthology.orchestra.run.smithi104.stdout:(6/23): librgw2-15.1.0-1904.g9eb9490.el8.x86_64 15 MB/s | 6.2 MB 00:00 2020-03-09T00:20:54.792 INFO:teuthology.orchestra.run.smithi104.stdout:(7/23): python3-ceph-argparse-15.1.0-1904.g9eb9 1.9 MB/s | 40 kB 00:00 2020-03-09T00:20:54.810 INFO:teuthology.orchestra.run.smithi104.stdout:(8/23): python3-ceph-common-15.1.0-1904.g9eb949 2.3 MB/s | 49 kB 00:00 2020-03-09T00:20:54.824 INFO:teuthology.orchestra.run.smithi148.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 372 kB/s | 47 kB 00:00 2020-03-09T00:20:54.845 INFO:teuthology.orchestra.run.smithi104.stdout:(9/23): python3-rados-15.1.0-1904.g9eb9490.el8. 9.9 MB/s | 351 kB 00:00 2020-03-09T00:20:54.868 INFO:teuthology.orchestra.run.smithi104.stdout:(10/23): python3-cephfs-15.1.0-1904.g9eb9490.el 2.1 MB/s | 162 kB 00:00 2020-03-09T00:20:54.883 INFO:teuthology.orchestra.run.smithi104.stdout:(11/23): python3-rbd-15.1.0-1904.g9eb9490.el8.x 8.0 MB/s | 309 kB 00:00 2020-03-09T00:20:54.933 INFO:teuthology.orchestra.run.smithi148.stdout:(16/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.8 MB/s | 315 kB 00:00 2020-03-09T00:20:54.949 INFO:teuthology.orchestra.run.smithi148.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 3.6 MB/s | 59 kB 00:00 2020-03-09T00:20:54.961 INFO:teuthology.orchestra.run.smithi104.stdout:(12/23): python3-rgw-15.1.0-1904.g9eb9490.el8.x 1.2 MB/s | 113 kB 00:00 2020-03-09T00:20:54.966 INFO:teuthology.orchestra.run.smithi148.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 4.5 MB/s | 75 kB 00:00 2020-03-09T00:20:54.971 INFO:teuthology.orchestra.run.smithi148.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 31 MB/s | 179 kB 00:00 2020-03-09T00:20:55.021 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:20:55.118 INFO:teuthology.orchestra.run.smithi104.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 1.5 MB/s | 353 kB 00:00 2020-03-09T00:20:55.165 INFO:teuthology.orchestra.run.smithi104.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2020-03-09T00:20:55.176 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-09T00:20:55.235 INFO:teuthology.orchestra.run.smithi104.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 374 kB/s | 44 kB 00:00 2020-03-09T00:20:55.291 INFO:teuthology.orchestra.run.smithi104.stdout:(16/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 372 kB/s | 47 kB 00:00 2020-03-09T00:20:55.329 INFO:teuthology.orchestra.run.smithi104.stdout:(17/23): gperftools-libs-2.7-6.el8.x86_64.rpm 3.3 MB/s | 315 kB 00:00 2020-03-09T00:20:55.346 INFO:teuthology.orchestra.run.smithi104.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 4.6 MB/s | 75 kB 00:00 2020-03-09T00:20:55.351 INFO:teuthology.orchestra.run.smithi104.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 32 MB/s | 179 kB 00:00 2020-03-09T00:20:55.352 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:20:55.366 INFO:teuthology.orchestra.run.smithi104.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 783 kB/s | 59 kB 00:00 2020-03-09T00:20:55.764 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:20:55.798 INFO:teuthology.orchestra.run.smithi050.stdout:(1/23): ceph-base-15.1.0-1904.g9eb9490.el8.x86_ 1.1 MB/s | 5.5 MB 00:04 2020-03-09T00:20:55.858 INFO:teuthology.orchestra.run.smithi050.stdout:(2/23): ceph-selinux-15.1.0-1904.g9eb9490.el8.x 413 kB/s | 25 kB 00:00 2020-03-09T00:20:55.902 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:20:55.982 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:20:55.995 INFO:teuthology.orchestra.run.smithi050.stdout:(3/23): libcephfs2-15.1.0-1904.g9eb9490.el8.x86 3.6 MB/s | 507 kB 00:00 2020-03-09T00:20:56.019 INFO:teuthology.orchestra.run.smithi148.stdout:(20/23): librados2-15.1.0-1904.g9eb9490.el8.x86 3.4 MB/s | 3.6 MB 00:01 2020-03-09T00:20:56.120 INFO:teuthology.orchestra.run.smithi050.stdout:(4/23): libradosstriper1-15.1.0-1904.g9eb9490.e 2.9 MB/s | 377 kB 00:00 2020-03-09T00:20:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:20:56.251 INFO:teuthology.orchestra.run.smithi104.stdout:(21/23): librbd1-15.1.0-1904.g9eb9490.el8.x86_6 2.2 MB/s | 1.9 MB 00:00 2020-03-09T00:20:56.251 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:20:56.441 INFO:teuthology.orchestra.run.smithi148.stdout:(21/23): librbd1-15.1.0-1904.g9eb9490.el8.x86_6 4.5 MB/s | 1.9 MB 00:00 2020-03-09T00:20:56.499 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:20:56.676 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:20:56.756 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:20:56.874 INFO:teuthology.orchestra.run.smithi050.stdout:(5/23): ceph-radosgw-15.1.0-1904.g9eb9490.el8.x 1.7 MB/s | 9.8 MB 00:05 2020-03-09T00:20:56.891 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-09T00:20:57.003 INFO:teuthology.orchestra.run.smithi050.stdout:(6/23): ceph-common-15.1.0-1904.g9eb9490.el8.x8 3.4 MB/s | 21 MB 00:06 2020-03-09T00:20:57.003 INFO:teuthology.orchestra.run.smithi050.stdout:(7/23): python3-ceph-argparse-15.1.0-1904.g9eb9 308 kB/s | 40 kB 00:00 2020-03-09T00:20:57.031 INFO:teuthology.orchestra.run.smithi050.stdout:(8/23): python3-ceph-common-15.1.0-1904.g9eb949 1.7 MB/s | 49 kB 00:00 2020-03-09T00:20:57.033 INFO:teuthology.orchestra.run.smithi050.stdout:(9/23): python3-cephfs-15.1.0-1904.g9eb9490.el8 5.4 MB/s | 162 kB 00:00 2020-03-09T00:20:57.035 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:20:57.072 INFO:teuthology.orchestra.run.smithi050.stdout:(10/23): python3-rados-15.1.0-1904.g9eb9490.el8 8.5 MB/s | 351 kB 00:00 2020-03-09T00:20:57.083 INFO:teuthology.orchestra.run.smithi050.stdout:(11/23): python3-rbd-15.1.0-1904.g9eb9490.el8.x 6.1 MB/s | 309 kB 00:00 2020-03-09T00:20:57.113 INFO:teuthology.orchestra.run.smithi050.stdout:(12/23): python3-rgw-15.1.0-1904.g9eb9490.el8.x 2.7 MB/s | 113 kB 00:00 2020-03-09T00:20:57.139 INFO:teuthology.orchestra.run.smithi050.stdout:(13/23): librgw2-15.1.0-1904.g9eb9490.el8.x86_6 6.1 MB/s | 6.2 MB 00:01 2020-03-09T00:20:57.202 INFO:teuthology.orchestra.run.smithi104.stdout:(22/23): ceph-radosgw-15.1.0-1904.g9eb9490.el8. 1.5 MB/s | 9.8 MB 00:06 2020-03-09T00:20:57.252 INFO:teuthology.orchestra.run.smithi104.stdout:(23/23): librados2-15.1.0-1904.g9eb9490.el8.x86 1.9 MB/s | 3.6 MB 00:01 2020-03-09T00:20:57.256 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:20:57.256 INFO:teuthology.orchestra.run.smithi104.stdout:Total 7.3 MB/s | 51 MB 00:06 2020-03-09T00:20:57.256 INFO:teuthology.orchestra.run.smithi050.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 375 kB/s | 44 kB 00:00 2020-03-09T00:20:57.261 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:20:57.280 INFO:teuthology.orchestra.run.smithi148.stdout:(22/23): ceph-radosgw-15.1.0-1904.g9eb9490.el8. 1.5 MB/s | 9.8 MB 00:06 2020-03-09T00:20:57.307 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:20:57.307 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:20:57.318 INFO:teuthology.orchestra.run.smithi050.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2020-03-09T00:20:57.319 INFO:teuthology.orchestra.run.smithi050.stdout:(16/23): librdkafka-0.11.4-1.el8.x86_64.rpm 1.5 MB/s | 353 kB 00:00 2020-03-09T00:20:57.381 INFO:teuthology.orchestra.run.smithi050.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 951 kB/s | 59 kB 00:00 2020-03-09T00:20:57.382 INFO:teuthology.orchestra.run.smithi050.stdout:(18/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 373 kB/s | 47 kB 00:00 2020-03-09T00:20:57.387 INFO:teuthology.orchestra.run.smithi050.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 31 MB/s | 179 kB 00:00 2020-03-09T00:20:57.412 INFO:teuthology.orchestra.run.smithi050.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.4 MB/s | 75 kB 00:00 2020-03-09T00:20:57.431 INFO:teuthology.orchestra.run.smithi050.stdout:(21/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.7 MB/s | 315 kB 00:00 2020-03-09T00:20:57.553 INFO:teuthology.orchestra.run.smithi050.stdout:(22/23): librbd1-15.1.0-1904.g9eb9490.el8.x86_6 14 MB/s | 1.9 MB 00:00 2020-03-09T00:20:57.624 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:20:57.625 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:20:57.653 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:20:57.653 INFO:teuthology.orchestra.run.smithi050.stdout:(23/23): librados2-15.1.0-1904.g9eb9490.el8.x86 13 MB/s | 3.6 MB 00:00 2020-03-09T00:20:57.655 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:20:57.656 INFO:teuthology.orchestra.run.smithi050.stdout:Total 7.4 MB/s | 51 MB 00:06 2020-03-09T00:20:57.661 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:20:57.708 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:20:57.747 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:20:57.760 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:20:57.858 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:20:58.040 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 10/25 2020-03-09T00:20:58.048 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:20:58.049 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:20:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:20:58.388 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:20:58.568 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:20:58.623 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:20:58.712 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:20:58.789 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-09T00:20:58.806 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:20:58.963 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-09T00:20:58.975 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:20:59.072 INFO:teuthology.orchestra.run.smithi148.stdout:(23/23): ceph-common-15.1.0-1904.g9eb9490.el8.x 2.5 MB/s | 21 MB 00:08 2020-03-09T00:20:59.075 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:20:59.076 INFO:teuthology.orchestra.run.smithi148.stdout:Total 5.9 MB/s | 51 MB 00:08 2020-03-09T00:20:59.081 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:20:59.112 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-09T00:20:59.127 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:20:59.128 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:20:59.173 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:20:59.275 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-09T00:20:59.365 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:20:59.367 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-09T00:20:59.432 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-09T00:20:59.435 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:20:59.435 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:20:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:20:59.542 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:20:59.593 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 19/25 2020-03-09T00:20:59.615 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:20:59.833 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:20:59.944 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:20:59.983 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:00.157 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:00.227 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:00.355 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:21:00.405 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:21:00.425 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:00.490 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:00.511 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:00.554 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-09T00:21:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-09T00:21:00.710 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:21:00.752 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:00.841 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:21:00.953 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:01.005 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:01.100 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-09T00:21:01.146 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-09T00:21:01.261 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:01.261 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:21:01.358 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:01.450 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:21:01.568 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:21:01.583 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:21:01.650 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:21:01.694 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:01.883 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 10/25 2020-03-09T00:21:01.917 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:21:01.918 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:02.069 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:21:02.094 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:02.177 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:21:02.183 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:02.245 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:21:02.249 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:21:02.317 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-09T00:21:02.416 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:21:02.435 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:21:02.455 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 10/25 2020-03-09T00:21:02.581 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:21:02.638 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:21:02.650 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:21:02.729 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:02.801 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:21:02.811 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-09T00:21:02.926 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:02.966 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-09T00:21:02.988 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:21:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-09T00:21:03.074 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-09T00:21:03.168 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:21:03.169 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:21:03.235 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 9/25 2020-03-09T00:21:03.239 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-09T00:21:03.240 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:21:03.400 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 19/25 2020-03-09T00:21:03.426 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-09T00:21:03.452 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 10/25 2020-03-09T00:21:03.564 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:03.564 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-09T00:21:03.605 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:21:03.719 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-09T00:21:03.781 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:21:03.797 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:03.862 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-09T00:21:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:03.968 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:21:04.017 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:04.027 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 19/25 2020-03-09T00:21:04.122 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:21:04.191 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 14/25 2020-03-09T00:21:04.347 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-09T00:21:04.510 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-09T00:21:04.659 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-09T00:21:04.833 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-09T00:21:04.994 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 19/25 2020-03-09T00:21:06.496 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:06.674 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:07.312 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:07.523 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:07.583 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:07.611 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:07.777 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:08.077 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:08.243 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:08.459 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:08.665 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:08.763 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:08.889 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 20/25 2020-03-09T00:21:09.105 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:09.193 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 21/25 2020-03-09T00:21:09.301 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:20.408 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:20.408 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2020-03-09T00:21:20.408 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2020-03-09T00:21:20.408 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2020-03-09T00:21:20.408 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /run 2020-03-09T00:21:20.408 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2020-03-09T00:21:20.409 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2020-03-09T00:21:20.409 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2020-03-09T00:21:20.409 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2020-03-09T00:21:20.409 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2020-03-09T00:21:20.409 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:21.464 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:21.580 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:21.580 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:21.791 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:21.791 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:22.431 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:22.431 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 1/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 7/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 8/25 2020-03-09T00:21:22.432 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 9/25 2020-03-09T00:21:22.433 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 10/25 2020-03-09T00:21:22.433 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:21:22.433 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:21:22.433 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:21:22.433 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-09T00:21:22.433 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-09T00:21:22.433 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-09T00:21:22.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 24/25 2020-03-09T00:21:23.237 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:23.237 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:23.237 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2020-03-09T00:21:23.238 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.238 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.238 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:23.238 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:21:23.238 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.238 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.238 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.239 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-09T00:21:23.240 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-09T00:21:23.241 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-09T00:21:23.241 INFO:teuthology.orchestra.run.smithi139.stdout: leveldb-1.21-2.el8.x86_64 2020-03-09T00:21:23.241 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:23.241 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:21:23.591 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-test 2020-03-09T00:21:23.978 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:23.978 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2020-03-09T00:21:23.978 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2020-03-09T00:21:23.978 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2020-03-09T00:21:23.978 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /run 2020-03-09T00:21:23.979 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2020-03-09T00:21:23.979 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2020-03-09T00:21:23.979 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2020-03-09T00:21:23.979 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2020-03-09T00:21:23.979 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2020-03-09T00:21:23.979 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:24.067 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:35 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:24.483 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:21:24.484 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:24.484 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2020-03-09T00:21:24.484 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:24.485 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:21:24.485 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 42 M 2020-03-09T00:21:24.485 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:21:24.485 INFO:teuthology.orchestra.run.smithi139.stdout: jq x86_64 1.5-12.el8 AppStream 161 k 2020-03-09T00:21:24.486 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma x86_64 6.8.2-1.el8 AppStream 188 k 2020-03-09T00:21:24.486 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.3.2-6.el8 AppStream 298 k 2020-03-09T00:21:24.486 INFO:teuthology.orchestra.run.smithi139.stdout: libxslt x86_64 1.1.32-3.el8 BaseOS 249 k 2020-03-09T00:21:24.486 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-09T00:21:24.486 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:24.487 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:21:24.487 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:24.487 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2020-03-09T00:21:24.487 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:24.488 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 43 M 2020-03-09T00:21:24.489 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 180 M 2020-03-09T00:21:24.489 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:21:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:25.070 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 745 kB/s | 161 kB 00:00 2020-03-09T00:21:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:25.106 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 745 kB/s | 188 kB 00:00 2020-03-09T00:21:25.297 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:25.297 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:25.309 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): libxslt-1.1.32-3.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2020-03-09T00:21:25.322 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): socat-1.7.3.2-6.el8.x86_64.rpm 1.2 MB/s | 298 kB 00:00 2020-03-09T00:21:25.386 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 898 kB/s | 69 kB 00:00 2020-03-09T00:21:25.579 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:25.579 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2020-03-09T00:21:25.579 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2020-03-09T00:21:25.579 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2020-03-09T00:21:25.580 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /run 2020-03-09T00:21:25.580 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2020-03-09T00:21:25.580 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2020-03-09T00:21:25.580 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2020-03-09T00:21:25.580 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2020-03-09T00:21:25.580 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /dev 2020-03-09T00:21:25.580 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:26.007 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:26.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 1/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 7/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 8/25 2020-03-09T00:21:26.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 9/25 2020-03-09T00:21:26.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 10/25 2020-03-09T00:21:26.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:21:26.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:21:26.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:21:26.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-09T00:21:26.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-09T00:21:26.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-09T00:21:26.010 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 24/25 2020-03-09T00:21:26.011 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:26.011 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2020-03-09T00:21:26.011 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2020-03-09T00:21:26.011 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2020-03-09T00:21:26.011 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /run 2020-03-09T00:21:26.012 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2020-03-09T00:21:26.012 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2020-03-09T00:21:26.012 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2020-03-09T00:21:26.012 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2020-03-09T00:21:26.012 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2020-03-09T00:21:26.012 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:26.623 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:26.648 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:26.648 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:26.648 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2020-03-09T00:21:26.648 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.648 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.648 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:26.648 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:21:26.649 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.649 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.649 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.649 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.649 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.649 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:26.650 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-09T00:21:26.651 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-09T00:21:26.651 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-09T00:21:26.651 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-09T00:21:26.651 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-09T00:21:26.651 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-09T00:21:26.651 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-09T00:21:26.652 INFO:teuthology.orchestra.run.smithi104.stdout: leveldb-1.21-2.el8.x86_64 2020-03-09T00:21:26.652 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:26.652 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:21:26.706 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:26.706 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:26.867 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:26.867 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:26.972 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2020-03-09T00:21:27.125 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): ceph-test-15.1.0-1904.g9eb9490.el8.x86_6 18 MB/s | 42 MB 00:02 2020-03-09T00:21:27.126 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:27.127 INFO:teuthology.orchestra.run.smithi139.stdout:Total 16 MB/s | 43 MB 00:02 2020-03-09T00:21:27.133 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:21:27.141 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:27.155 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:21:27.155 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:21:27.220 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 23/25 2020-03-09T00:21:27.220 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:27.378 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:38 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:27.418 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-09T00:21:27.418 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:27.436 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:21:27.437 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:21:27.529 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:27.529 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 1/25 2020-03-09T00:21:27.529 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2/25 2020-03-09T00:21:27.529 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:27.529 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:21:27.529 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:27.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:27.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 7/25 2020-03-09T00:21:27.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 8/25 2020-03-09T00:21:27.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 9/25 2020-03-09T00:21:27.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 10/25 2020-03-09T00:21:27.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:21:27.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:21:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:21:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-09T00:21:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-09T00:21:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-09T00:21:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-09T00:21:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-09T00:21:27.531 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-09T00:21:27.532 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-09T00:21:27.532 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-09T00:21:27.532 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:27.532 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-09T00:21:27.532 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 24/25 2020-03-09T00:21:27.735 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:21:27.736 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:27.736 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2020-03-09T00:21:27.736 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:27.736 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:21:27.736 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 42 M 2020-03-09T00:21:27.737 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:21:27.737 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.5-12.el8 AppStream 161 k 2020-03-09T00:21:27.737 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.8.2-1.el8 AppStream 188 k 2020-03-09T00:21:27.737 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.3.2-6.el8 AppStream 298 k 2020-03-09T00:21:27.737 INFO:teuthology.orchestra.run.smithi104.stdout: libxslt x86_64 1.1.32-3.el8 BaseOS 249 k 2020-03-09T00:21:27.737 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-09T00:21:27.737 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:27.738 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:21:27.738 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:27.738 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2020-03-09T00:21:27.738 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:27.739 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 43 M 2020-03-09T00:21:27.739 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 180 M 2020-03-09T00:21:27.739 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:21:27.881 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:21:28.071 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:28.071 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 1/25 2020-03-09T00:21:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2/25 2020-03-09T00:21:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 3/25 2020-03-09T00:21:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 4/25 2020-03-09T00:21:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 5/25 2020-03-09T00:21:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 6/25 2020-03-09T00:21:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 7/25 2020-03-09T00:21:28.072 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x 8/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86 9/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 10/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 11/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 12/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 13/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-09T00:21:28.073 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-09T00:21:28.074 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-09T00:21:28.074 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-09T00:21:28.074 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-09T00:21:28.074 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-09T00:21:28.074 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-09T00:21:28.074 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 22/25 2020-03-09T00:21:28.074 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-09T00:21:28.075 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 24/25 2020-03-09T00:21:28.101 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-09T00:21:28.133 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 915 kB/s | 161 kB 00:00 2020-03-09T00:21:28.181 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 838 kB/s | 188 kB 00:00 2020-03-09T00:21:28.258 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-09T00:21:28.362 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:28.362 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:28.362 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2020-03-09T00:21:28.362 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.363 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.363 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:28.363 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:21:28.363 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.363 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.363 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.364 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.365 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.365 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.365 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.365 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-09T00:21:28.366 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-09T00:21:28.366 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-09T00:21:28.366 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-09T00:21:28.366 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-09T00:21:28.366 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-09T00:21:28.366 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-09T00:21:28.366 INFO:teuthology.orchestra.run.smithi148.stdout: leveldb-1.21-2.el8.x86_64 2020-03-09T00:21:28.367 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:28.367 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:21:28.368 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): libxslt-1.1.32-3.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2020-03-09T00:21:28.389 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): socat-1.7.3.2-6.el8.x86_64.rpm 1.1 MB/s | 298 kB 00:00 2020-03-09T00:21:28.440 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-09T00:21:28.444 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 911 kB/s | 69 kB 00:00 2020-03-09T00:21:28.579 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:28.657 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:28.667 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-test 2020-03-09T00:21:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-09T00:21:28.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:28.836 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2020-03-09T00:21:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:28.837 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:21:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.837 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-09T00:21:28.839 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-09T00:21:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-09T00:21:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-09T00:21:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-09T00:21:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-09T00:21:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-09T00:21:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: leveldb-1.21-2.el8.x86_64 2020-03-09T00:21:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:28.842 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:21:29.078 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:40 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:29.190 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2020-03-09T00:21:29.447 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:21:29.448 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:29.448 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2020-03-09T00:21:29.449 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:29.449 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:21:29.449 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 42 M 2020-03-09T00:21:29.449 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:21:29.449 INFO:teuthology.orchestra.run.smithi148.stdout: jq x86_64 1.5-12.el8 AppStream 161 k 2020-03-09T00:21:29.449 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma x86_64 6.8.2-1.el8 AppStream 188 k 2020-03-09T00:21:29.449 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.3.2-6.el8 AppStream 298 k 2020-03-09T00:21:29.450 INFO:teuthology.orchestra.run.smithi148.stdout: libxslt x86_64 1.1.32-3.el8 BaseOS 249 k 2020-03-09T00:21:29.450 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-09T00:21:29.450 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:29.450 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:21:29.450 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:29.450 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2020-03-09T00:21:29.450 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:29.451 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 43 M 2020-03-09T00:21:29.451 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 180 M 2020-03-09T00:21:29.452 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:21:29.622 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:40 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:29.877 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): ceph-test-15.1.0-1904.g9eb9490.el8.x86_6 22 MB/s | 42 MB 00:01 2020-03-09T00:21:29.878 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:29.878 INFO:teuthology.orchestra.run.smithi104.stdout:Total 20 MB/s | 43 MB 00:02 2020-03-09T00:21:29.883 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:21:29.904 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:21:29.904 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:21:30.019 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:21:30.019 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:30.019 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2020-03-09T00:21:30.020 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:30.020 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:21:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 42 M 2020-03-09T00:21:30.020 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:21:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: jq x86_64 1.5-12.el8 AppStream 161 k 2020-03-09T00:21:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma x86_64 6.8.2-1.el8 AppStream 188 k 2020-03-09T00:21:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.3.2-6.el8 AppStream 298 k 2020-03-09T00:21:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: libxslt x86_64 1.1.32-3.el8 BaseOS 249 k 2020-03-09T00:21:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-09T00:21:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:30.021 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:21:30.021 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:30.021 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2020-03-09T00:21:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:30.022 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 43 M 2020-03-09T00:21:30.022 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 180 M 2020-03-09T00:21:30.022 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:21:30.272 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:21:30.273 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:21:30.460 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 918 kB/s | 161 kB 00:00 2020-03-09T00:21:30.509 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 834 kB/s | 188 kB 00:00 2020-03-09T00:21:30.697 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): libxslt-1.1.32-3.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2020-03-09T00:21:30.731 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): oniguruma-6.8.2-1.el8.x86_64.rpm 919 kB/s | 188 kB 00:00 2020-03-09T00:21:30.747 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): socat-1.7.3.2-6.el8.x86_64.rpm 1.0 MB/s | 298 kB 00:00 2020-03-09T00:21:30.753 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 712 kB/s | 161 kB 00:00 2020-03-09T00:21:30.759 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-03-09T00:21:30.811 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:21:30.942 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): libxslt-1.1.32-3.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2020-03-09T00:21:30.988 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): socat-1.7.3.2-6.el8.x86_64.rpm 1.1 MB/s | 298 kB 00:00 2020-03-09T00:21:31.018 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 904 kB/s | 69 kB 00:00 2020-03-09T00:21:31.022 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-09T00:21:31.200 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-09T00:21:31.362 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-09T00:21:31.526 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:31.590 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:32.453 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): ceph-test-15.1.0-1904.g9eb9490.el8.x86_6 19 MB/s | 42 MB 00:02 2020-03-09T00:21:32.455 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:32.455 INFO:teuthology.orchestra.run.smithi050.stdout:Total 18 MB/s | 43 MB 00:02 2020-03-09T00:21:32.462 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:21:32.486 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:21:32.486 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:21:32.619 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-09T00:21:32.755 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:21:32.756 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:21:32.817 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:33.031 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:33.031 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 1/6 2020-03-09T00:21:33.031 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-09T00:21:33.031 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-09T00:21:33.031 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-09T00:21:33.031 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-09T00:21:33.254 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:21:33.396 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-09T00:21:33.396 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:33.397 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:21:33.397 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-09T00:21:33.397 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-09T00:21:33.397 INFO:teuthology.orchestra.run.smithi139.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-09T00:21:33.397 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:33.397 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:21:33.491 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-09T00:21:33.658 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-09T00:21:33.669 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph 2020-03-09T00:21:33.841 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-09T00:21:33.933 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): ceph-test-15.1.0-1904.g9eb9490.el8.x86_6 12 MB/s | 42 MB 00:03 2020-03-09T00:21:33.935 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:33.935 INFO:teuthology.orchestra.run.smithi148.stdout:Total 9.5 MB/s | 43 MB 00:04 2020-03-09T00:21:33.942 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:21:33.964 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:21:33.964 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:21:33.982 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:34.060 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:34.063 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:45 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:34.226 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:21:34.227 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:21:34.450 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:21:34.454 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:34.454 INFO:teuthology.orchestra.run.smithi139.stdout: Package ArchVersion Repository Size 2020-03-09T00:21:34.454 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:34.454 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:21:34.454 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_642:15.1.0-1904.g9eb9490.el8 ceph 6.4 k 2020-03-09T00:21:34.455 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:21:34.455 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_642:15.1.0-1904.g9eb9490.el8 ceph 2.0 M 2020-03-09T00:21:34.455 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_642:15.1.0-1904.g9eb9490.el8 ceph 1.4 M 2020-03-09T00:21:34.455 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_642:15.1.0-1904.g9eb9490.el8 ceph 4.4 M 2020-03-09T00:21:34.455 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_642:15.1.0-1904.g9eb9490.el8 ceph 17 M 2020-03-09T00:21:34.455 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 165 k 2020-03-09T00:21:34.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-lxml x86_644.2.3-1.el8 AppStream 1.5 M 2020-03-09T00:21:34.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch1.0.6-13.el8 AppStream 157 k 2020-03-09T00:21:34.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_640.23-19.el8 AppStream 39 k 2020-03-09T00:21:34.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch2017.2-9.el8 AppStream 54 k 2020-03-09T00:21:34.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webencodings noarch0.5.1-6.el8 AppStream 27 k 2020-03-09T00:21:34.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch0.12.2-4.el8 AppStream 457 k 2020-03-09T00:21:34.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-chardet noarch3.0.4-7.el8 BaseOS 195 k 2020-03-09T00:21:34.456 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pysocks noarch1.6.8-3.el8 BaseOS 34 k 2020-03-09T00:21:34.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch2.20.0-2.1.el8_1 BaseOS 123 k 2020-03-09T00:21:34.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch1.24.2-2.el8 BaseOS 176 k 2020-03-09T00:21:34.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-09T00:21:34.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-09T00:21:34.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-09T00:21:34.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-09T00:21:34.457 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-09T00:21:34.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-09T00:21:34.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-09T00:21:34.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-09T00:21:34.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-09T00:21:34.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-09T00:21:34.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-09T00:21:34.458 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2020-03-09T00:21:34.459 INFO:teuthology.orchestra.run.smithi139.stdout: python3-html5lib noarch1:0.999999999-6.el8 AppStream 214 k 2020-03-09T00:21:34.460 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-09T00:21:34.460 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:34.460 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:21:34.460 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:34.460 INFO:teuthology.orchestra.run.smithi139.stdout:Install 36 Packages 2020-03-09T00:21:34.460 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 29 M 2020-03-09T00:21:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 103 M 2020-03-09T00:21:34.461 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:21:34.704 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:21:34.842 INFO:teuthology.orchestra.run.smithi139.stdout:(1/36): ceph-15.1.0-1904.g9eb9490.el8.x86_64.rp 63 kB/s | 6.4 kB 00:00 2020-03-09T00:21:34.879 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-09T00:21:35.074 INFO:teuthology.orchestra.run.smithi139.stdout:(2/36): ceph-mds-15.1.0-1904.g9eb9490.el8.x86_6 6.1 MB/s | 2.0 MB 00:00 2020-03-09T00:21:35.075 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-09T00:21:35.139 INFO:teuthology.orchestra.run.smithi139.stdout:(3/36): ceph-mgr-15.1.0-1904.g9eb9490.el8.x86_6 3.5 MB/s | 1.4 MB 00:00 2020-03-09T00:21:35.217 INFO:teuthology.orchestra.run.smithi139.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1904.g9eb9 2.1 MB/s | 165 kB 00:00 2020-03-09T00:21:35.241 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-09T00:21:35.380 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:35.460 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-09T00:21:35.464 INFO:teuthology.orchestra.run.smithi139.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 865 kB/s | 214 kB 00:00 2020-03-09T00:21:35.578 INFO:teuthology.orchestra.run.smithi139.stdout:(6/36): ceph-mon-15.1.0-1904.g9eb9490.el8.x86_6 5.9 MB/s | 4.4 MB 00:00 2020-03-09T00:21:35.685 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-09T00:21:35.753 INFO:teuthology.orchestra.run.smithi139.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 898 kB/s | 157 kB 00:00 2020-03-09T00:21:35.895 INFO:teuthology.orchestra.run.smithi139.stdout:(8/36): ceph-osd-15.1.0-1904.g9eb9490.el8.x86_6 20 MB/s | 17 MB 00:00 2020-03-09T00:21:35.988 INFO:teuthology.orchestra.run.smithi139.stdout:(9/36): python3-markupsafe-0.23-19.el8.x86_64.r 165 kB/s | 39 kB 00:00 2020-03-09T00:21:35.994 INFO:teuthology.orchestra.run.smithi139.stdout:(10/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 2.8 MB/s | 1.5 MB 00:00 2020-03-09T00:21:36.036 INFO:teuthology.orchestra.run.smithi139.stdout:(11/36): python3-pytz-2017.2-9.el8.noarch.rpm 385 kB/s | 54 kB 00:00 2020-03-09T00:21:36.091 INFO:teuthology.orchestra.run.smithi139.stdout:(12/36): python3-webencodings-0.5.1-6.el8.noarc 262 kB/s | 27 kB 00:00 2020-03-09T00:21:36.192 INFO:teuthology.orchestra.run.smithi139.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 339 kB/s | 34 kB 00:00 2020-03-09T00:21:36.223 INFO:teuthology.orchestra.run.smithi139.stdout:(14/36): python3-chardet-3.0.4-7.el8.noarch.rpm 1.0 MB/s | 195 kB 00:00 2020-03-09T00:21:36.255 INFO:teuthology.orchestra.run.smithi139.stdout:(15/36): python3-requests-2.20.0-2.1.el8_1.noar 1.9 MB/s | 123 kB 00:00 2020-03-09T00:21:36.286 INFO:teuthology.orchestra.run.smithi139.stdout:(16/36): python3-urllib3-1.24.2-2.el8.noarch.rp 2.7 MB/s | 176 kB 00:00 2020-03-09T00:21:36.307 INFO:teuthology.orchestra.run.smithi139.stdout:(17/36): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2020-03-09T00:21:36.321 INFO:teuthology.orchestra.run.smithi139.stdout:(18/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.2 MB/s | 145 kB 00:00 2020-03-09T00:21:36.344 INFO:teuthology.orchestra.run.smithi139.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 794 kB/s | 18 kB 00:00 2020-03-09T00:21:36.363 INFO:teuthology.orchestra.run.smithi139.stdout:(20/36): python3-cherrypy-18.4.0-1.el8.noarch.r 4.9 MB/s | 385 kB 00:00 2020-03-09T00:21:36.370 INFO:teuthology.orchestra.run.smithi139.stdout:(21/36): python3-jaraco-6.2-6.el8.noarch.rpm 184 kB/s | 11 kB 00:00 2020-03-09T00:21:36.374 INFO:teuthology.orchestra.run.smithi139.stdout:(22/36): python3-more-itertools-7.2.0-3.el8.noa 5.2 MB/s | 59 kB 00:00 2020-03-09T00:21:36.379 INFO:teuthology.orchestra.run.smithi139.stdout:(23/36): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2020-03-09T00:21:36.383 INFO:teuthology.orchestra.run.smithi139.stdout:(24/36): python3-portend-2.6-1.el8.noarch.rpm 1.7 MB/s | 16 kB 00:00 2020-03-09T00:21:36.389 INFO:teuthology.orchestra.run.smithi139.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 1.9 MB/s | 19 kB 00:00 2020-03-09T00:21:36.393 INFO:teuthology.orchestra.run.smithi139.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.4 MB/s | 24 kB 00:00 2020-03-09T00:21:36.400 INFO:teuthology.orchestra.run.smithi139.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 2.7 MB/s | 30 kB 00:00 2020-03-09T00:21:36.404 INFO:teuthology.orchestra.run.smithi139.stdout:(28/36): python3-trustme-0.5.2-4.el8.noarch.rpm 2.4 MB/s | 27 kB 00:00 2020-03-09T00:21:36.406 INFO:teuthology.orchestra.run.smithi139.stdout:(29/36): python3-pecan-1.3.2-9.el8.noarch.rpm 7.8 MB/s | 283 kB 00:00 2020-03-09T00:21:36.410 INFO:teuthology.orchestra.run.smithi139.stdout:(30/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.3 MB/s | 23 kB 00:00 2020-03-09T00:21:36.467 INFO:teuthology.orchestra.run.smithi139.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 712 kB/s | 40 kB 00:00 2020-03-09T00:21:36.468 INFO:teuthology.orchestra.run.smithi139.stdout:(32/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 690 kB/s | 44 kB 00:00 2020-03-09T00:21:36.490 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:36.501 INFO:teuthology.orchestra.run.smithi139.stdout:(33/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-09T00:21:36.515 INFO:teuthology.orchestra.run.smithi139.stdout:(34/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 5.2 MB/s | 251 kB 00:00 2020-03-09T00:21:36.521 INFO:teuthology.orchestra.run.smithi139.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.2 MB/s | 85 kB 00:00 2020-03-09T00:21:36.524 INFO:teuthology.orchestra.run.smithi139.stdout:(36/36): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2020-03-09T00:21:36.529 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:36.529 INFO:teuthology.orchestra.run.smithi139.stdout:Total 14 MB/s | 29 MB 00:02 2020-03-09T00:21:36.551 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:21:36.591 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:21:36.592 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:21:36.720 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:36.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 1/6 2020-03-09T00:21:36.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-09T00:21:36.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-09T00:21:36.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-09T00:21:36.721 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-09T00:21:36.920 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:21:36.921 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:21:37.012 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-09T00:21:37.012 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:37.012 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:21:37.012 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-09T00:21:37.012 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-09T00:21:37.012 INFO:teuthology.orchestra.run.smithi104.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-09T00:21:37.013 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:37.013 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:21:37.284 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2020-03-09T00:21:37.678 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:48 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:38.104 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:21:38.108 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:38.108 INFO:teuthology.orchestra.run.smithi104.stdout: Package ArchVersion Repository Size 2020-03-09T00:21:38.108 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:38.108 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:21:38.109 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_642:15.1.0-1904.g9eb9490.el8 ceph 6.4 k 2020-03-09T00:21:38.109 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:21:38.109 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_642:15.1.0-1904.g9eb9490.el8 ceph 2.0 M 2020-03-09T00:21:38.109 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_642:15.1.0-1904.g9eb9490.el8 ceph 1.4 M 2020-03-09T00:21:38.109 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_642:15.1.0-1904.g9eb9490.el8 ceph 4.4 M 2020-03-09T00:21:38.109 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_642:15.1.0-1904.g9eb9490.el8 ceph 17 M 2020-03-09T00:21:38.110 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 165 k 2020-03-09T00:21:38.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-lxml x86_644.2.3-1.el8 AppStream 1.5 M 2020-03-09T00:21:38.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch1.0.6-13.el8 AppStream 157 k 2020-03-09T00:21:38.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_640.23-19.el8 AppStream 39 k 2020-03-09T00:21:38.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch2017.2-9.el8 AppStream 54 k 2020-03-09T00:21:38.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webencodings noarch0.5.1-6.el8 AppStream 27 k 2020-03-09T00:21:38.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch0.12.2-4.el8 AppStream 457 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-chardet noarch3.0.4-7.el8 BaseOS 195 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pysocks noarch1.6.8-3.el8 BaseOS 34 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch2.20.0-2.1.el8_1 BaseOS 123 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch1.24.2-2.el8 BaseOS 176 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-09T00:21:38.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-09T00:21:38.112 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-09T00:21:38.112 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-09T00:21:38.112 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-09T00:21:38.112 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-09T00:21:38.112 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-09T00:21:38.112 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-09T00:21:38.112 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-09T00:21:38.113 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2020-03-09T00:21:38.114 INFO:teuthology.orchestra.run.smithi104.stdout: python3-html5lib noarch1:0.999999999-6.el8 AppStream 214 k 2020-03-09T00:21:38.114 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-09T00:21:38.114 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:21:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:Install 36 Packages 2020-03-09T00:21:38.114 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:38.117 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 29 M 2020-03-09T00:21:38.117 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 103 M 2020-03-09T00:21:38.117 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:21:38.277 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:21:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-09T00:21:38.456 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-09T00:21:38.550 INFO:teuthology.orchestra.run.smithi104.stdout:(1/36): ceph-15.1.0-1904.g9eb9490.el8.x86_64.rp 72 kB/s | 6.4 kB 00:00 2020-03-09T00:21:38.741 INFO:teuthology.orchestra.run.smithi104.stdout:(2/36): ceph-mgr-15.1.0-1904.g9eb9490.el8.x86_6 5.0 MB/s | 1.4 MB 00:00 2020-03-09T00:21:38.774 INFO:teuthology.orchestra.run.smithi104.stdout:(3/36): ceph-mds-15.1.0-1904.g9eb9490.el8.x86_6 6.5 MB/s | 2.0 MB 00:00 2020-03-09T00:21:38.783 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-09T00:21:38.822 INFO:teuthology.orchestra.run.smithi104.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1904.g9eb9 3.4 MB/s | 165 kB 00:00 2020-03-09T00:21:39.008 INFO:teuthology.orchestra.run.smithi104.stdout:(5/36): ceph-mon-15.1.0-1904.g9eb9490.el8.x86_6 9.5 MB/s | 4.4 MB 00:00 2020-03-09T00:21:39.038 INFO:teuthology.orchestra.run.smithi104.stdout:(6/36): python3-html5lib-0.999999999-6.el8.noar 991 kB/s | 214 kB 00:00 2020-03-09T00:21:39.054 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-09T00:21:39.212 INFO:teuthology.orchestra.run.smithi104.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 900 kB/s | 157 kB 00:00 2020-03-09T00:21:39.245 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-09T00:21:39.330 INFO:teuthology.orchestra.run.smithi104.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 331 kB/s | 39 kB 00:00 2020-03-09T00:21:39.362 INFO:teuthology.orchestra.run.smithi104.stdout:(9/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 4.3 MB/s | 1.5 MB 00:00 2020-03-09T00:21:39.380 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:39.444 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-09T00:21:39.450 INFO:teuthology.orchestra.run.smithi104.stdout:(10/36): python3-pytz-2017.2-9.el8.noarch.rpm 451 kB/s | 54 kB 00:00 2020-03-09T00:21:39.469 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-09T00:21:39.548 INFO:teuthology.orchestra.run.smithi104.stdout:(11/36): ceph-osd-15.1.0-1904.g9eb9490.el8.x86_ 21 MB/s | 17 MB 00:00 2020-03-09T00:21:39.549 INFO:teuthology.orchestra.run.smithi104.stdout:(12/36): python3-webencodings-0.5.1-6.el8.noarc 144 kB/s | 27 kB 00:00 2020-03-09T00:21:39.627 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:39.643 INFO:teuthology.orchestra.run.smithi104.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 365 kB/s | 34 kB 00:00 2020-03-09T00:21:39.669 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-09T00:21:39.706 INFO:teuthology.orchestra.run.smithi104.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 1.9 MB/s | 123 kB 00:00 2020-03-09T00:21:39.725 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:39.725 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 1/6 2020-03-09T00:21:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-09T00:21:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-09T00:21:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-09T00:21:39.726 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-09T00:21:39.736 INFO:teuthology.orchestra.run.smithi104.stdout:(15/36): python3-chardet-3.0.4-7.el8.noarch.rpm 1.0 MB/s | 195 kB 00:00 2020-03-09T00:21:39.769 INFO:teuthology.orchestra.run.smithi104.stdout:(16/36): python3-urllib3-1.24.2-2.el8.noarch.rp 2.7 MB/s | 176 kB 00:00 2020-03-09T00:21:39.783 INFO:teuthology.orchestra.run.smithi104.stdout:(17/36): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2020-03-09T00:21:39.796 INFO:teuthology.orchestra.run.smithi104.stdout:(18/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.3 MB/s | 145 kB 00:00 2020-03-09T00:21:39.806 INFO:teuthology.orchestra.run.smithi104.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.0 MB/s | 18 kB 00:00 2020-03-09T00:21:39.817 INFO:teuthology.orchestra.run.smithi104.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 4.1 MB/s | 49 kB 00:00 2020-03-09T00:21:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 6/6 2020-03-09T00:21:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 1/6 2020-03-09T00:21:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-09T00:21:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-09T00:21:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-09T00:21:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-09T00:21:39.826 INFO:teuthology.orchestra.run.smithi104.stdout:(21/36): python3-jaraco-6.2-6.el8.noarch.rpm 276 kB/s | 11 kB 00:00 2020-03-09T00:21:39.827 INFO:teuthology.orchestra.run.smithi104.stdout:(22/36): python3-more-itertools-7.2.0-3.el8.noa 6.1 MB/s | 59 kB 00:00 2020-03-09T00:21:39.834 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-09T00:21:39.836 INFO:teuthology.orchestra.run.smithi104.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.8 MB/s | 16 kB 00:00 2020-03-09T00:21:39.842 INFO:teuthology.orchestra.run.smithi104.stdout:(24/36): python3-cherrypy-18.4.0-1.el8.noarch.r 5.2 MB/s | 385 kB 00:00 2020-03-09T00:21:39.844 INFO:teuthology.orchestra.run.smithi104.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 2.2 MB/s | 19 kB 00:00 2020-03-09T00:21:39.851 INFO:teuthology.orchestra.run.smithi104.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.7 MB/s | 24 kB 00:00 2020-03-09T00:21:39.853 INFO:teuthology.orchestra.run.smithi104.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.5 MB/s | 30 kB 00:00 2020-03-09T00:21:39.858 INFO:teuthology.orchestra.run.smithi104.stdout:(28/36): python3-pecan-1.3.2-9.el8.noarch.rpm 8.5 MB/s | 283 kB 00:00 2020-03-09T00:21:39.860 INFO:teuthology.orchestra.run.smithi104.stdout:(29/36): python3-trustme-0.5.2-4.el8.noarch.rpm 3.0 MB/s | 27 kB 00:00 2020-03-09T00:21:39.861 INFO:teuthology.orchestra.run.smithi104.stdout:(30/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.8 MB/s | 23 kB 00:00 2020-03-09T00:21:39.909 INFO:teuthology.orchestra.run.smithi104.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 861 kB/s | 40 kB 00:00 2020-03-09T00:21:39.921 INFO:teuthology.orchestra.run.smithi104.stdout:(32/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 692 kB/s | 44 kB 00:00 2020-03-09T00:21:39.954 INFO:teuthology.orchestra.run.smithi104.stdout:(33/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-09T00:21:39.955 INFO:teuthology.orchestra.run.smithi104.stdout:(34/36): python3-waitress-1.2.1-2.el8.1.noarch. 5.1 MB/s | 241 kB 00:00 2020-03-09T00:21:39.957 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-09T00:21:39.974 INFO:teuthology.orchestra.run.smithi104.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.2 MB/s | 85 kB 00:00 2020-03-09T00:21:39.983 INFO:teuthology.orchestra.run.smithi104.stdout:(36/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.0 MB/s | 251 kB 00:00 2020-03-09T00:21:39.987 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:39.987 INFO:teuthology.orchestra.run.smithi104.stdout:Total 16 MB/s | 29 MB 00:01 2020-03-09T00:21:40.007 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:21:40.046 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:21:40.047 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:21:40.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-09T00:21:40.102 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:40.102 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:21:40.102 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-09T00:21:40.102 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-09T00:21:40.102 INFO:teuthology.orchestra.run.smithi050.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-09T00:21:40.102 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:40.103 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:21:40.147 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-09T00:21:40.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-09T00:21:40.151 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:40.151 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:21:40.151 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:15.1.0-1904.g9eb9490.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-09T00:21:40.151 INFO:teuthology.orchestra.run.smithi148.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-09T00:21:40.151 INFO:teuthology.orchestra.run.smithi148.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-09T00:21:40.152 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:40.152 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:21:40.330 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-09T00:21:40.391 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:21:40.391 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:21:40.413 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2020-03-09T00:21:40.430 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph 2020-03-09T00:21:40.500 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-09T00:21:40.677 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-09T00:21:40.827 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:51 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:40.835 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:51 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:40.901 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-09T00:21:41.116 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-09T00:21:41.247 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:21:41.251 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:41.251 INFO:teuthology.orchestra.run.smithi148.stdout: Package ArchVersion Repository Size 2020-03-09T00:21:41.251 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:41.251 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:21:41.252 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_642:15.1.0-1904.g9eb9490.el8 ceph 6.4 k 2020-03-09T00:21:41.252 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:21:41.252 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_642:15.1.0-1904.g9eb9490.el8 ceph 2.0 M 2020-03-09T00:21:41.252 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_642:15.1.0-1904.g9eb9490.el8 ceph 1.4 M 2020-03-09T00:21:41.252 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_642:15.1.0-1904.g9eb9490.el8 ceph 4.4 M 2020-03-09T00:21:41.252 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_642:15.1.0-1904.g9eb9490.el8 ceph 17 M 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 165 k 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: python3-lxml x86_644.2.3-1.el8 AppStream 1.5 M 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch1.0.6-13.el8 AppStream 157 k 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_640.23-19.el8 AppStream 39 k 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch2017.2-9.el8 AppStream 54 k 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webencodings noarch0.5.1-6.el8 AppStream 27 k 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch0.12.2-4.el8 AppStream 457 k 2020-03-09T00:21:41.253 INFO:teuthology.orchestra.run.smithi148.stdout: python3-chardet noarch3.0.4-7.el8 BaseOS 195 k 2020-03-09T00:21:41.254 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pysocks noarch1.6.8-3.el8 BaseOS 34 k 2020-03-09T00:21:41.254 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests noarch2.20.0-2.1.el8_1 BaseOS 123 k 2020-03-09T00:21:41.254 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3 noarch1.24.2-2.el8 BaseOS 176 k 2020-03-09T00:21:41.254 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-09T00:21:41.254 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-09T00:21:41.254 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-09T00:21:41.254 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-09T00:21:41.255 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-09T00:21:41.256 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-09T00:21:41.256 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-09T00:21:41.256 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-09T00:21:41.256 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-09T00:21:41.256 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-09T00:21:41.256 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-09T00:21:41.256 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2020-03-09T00:21:41.257 INFO:teuthology.orchestra.run.smithi148.stdout: python3-html5lib noarch1:0.999999999-6.el8 AppStream 214 k 2020-03-09T00:21:41.257 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-09T00:21:41.257 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:41.257 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:21:41.257 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:41.257 INFO:teuthology.orchestra.run.smithi148.stdout:Install 36 Packages 2020-03-09T00:21:41.258 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:41.258 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 29 M 2020-03-09T00:21:41.258 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 103 M 2020-03-09T00:21:41.258 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:21:41.261 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:21:41.265 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout: Package ArchVersion Repository Size 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_642:15.1.0-1904.g9eb9490.el8 ceph 6.4 k 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_642:15.1.0-1904.g9eb9490.el8 ceph 2.0 M 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_642:15.1.0-1904.g9eb9490.el8 ceph 1.4 M 2020-03-09T00:21:41.266 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_642:15.1.0-1904.g9eb9490.el8 ceph 4.4 M 2020-03-09T00:21:41.267 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_642:15.1.0-1904.g9eb9490.el8 ceph 17 M 2020-03-09T00:21:41.267 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 165 k 2020-03-09T00:21:41.267 INFO:teuthology.orchestra.run.smithi050.stdout: python3-lxml x86_644.2.3-1.el8 AppStream 1.5 M 2020-03-09T00:21:41.267 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch1.0.6-13.el8 AppStream 157 k 2020-03-09T00:21:41.267 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_640.23-19.el8 AppStream 39 k 2020-03-09T00:21:41.267 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch2017.2-9.el8 AppStream 54 k 2020-03-09T00:21:41.267 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webencodings noarch0.5.1-6.el8 AppStream 27 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch0.12.2-4.el8 AppStream 457 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-chardet noarch3.0.4-7.el8 BaseOS 195 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pysocks noarch1.6.8-3.el8 BaseOS 34 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch2.20.0-2.1.el8_1 BaseOS 123 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch1.24.2-2.el8 BaseOS 176 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-09T00:21:41.268 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-09T00:21:41.269 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-09T00:21:41.270 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2020-03-09T00:21:41.271 INFO:teuthology.orchestra.run.smithi050.stdout: python3-html5lib noarch1:0.999999999-6.el8 AppStream 214 k 2020-03-09T00:21:41.271 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-09T00:21:41.271 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:41.271 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:21:41.271 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:41.271 INFO:teuthology.orchestra.run.smithi050.stdout:Install 36 Packages 2020-03-09T00:21:41.271 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:41.272 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 29 M 2020-03-09T00:21:41.272 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 103 M 2020-03-09T00:21:41.272 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:21:41.405 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-09T00:21:41.609 INFO:teuthology.orchestra.run.smithi050.stdout:(1/36): ceph-15.1.0-1904.g9eb9490.el8.x86_64.rp 69 kB/s | 6.4 kB 00:00 2020-03-09T00:21:41.624 INFO:teuthology.orchestra.run.smithi148.stdout:(1/36): ceph-15.1.0-1904.g9eb9490.el8.x86_64.rp 72 kB/s | 6.4 kB 00:00 2020-03-09T00:21:41.633 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-09T00:21:41.887 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-09T00:21:41.898 INFO:teuthology.orchestra.run.smithi050.stdout:(2/36): ceph-mgr-15.1.0-1904.g9eb9490.el8.x86_6 3.7 MB/s | 1.4 MB 00:00 2020-03-09T00:21:41.934 INFO:teuthology.orchestra.run.smithi050.stdout:(3/36): ceph-mds-15.1.0-1904.g9eb9490.el8.x86_6 4.9 MB/s | 2.0 MB 00:00 2020-03-09T00:21:41.999 INFO:teuthology.orchestra.run.smithi050.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1904.g9eb9 2.5 MB/s | 165 kB 00:00 2020-03-09T00:21:42.005 INFO:teuthology.orchestra.run.smithi148.stdout:(2/36): ceph-mgr-15.1.0-1904.g9eb9490.el8.x86_6 3.0 MB/s | 1.4 MB 00:00 2020-03-09T00:21:42.033 INFO:teuthology.orchestra.run.smithi148.stdout:(3/36): ceph-mds-15.1.0-1904.g9eb9490.el8.x86_6 4.1 MB/s | 2.0 MB 00:00 2020-03-09T00:21:42.079 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-09T00:21:42.088 INFO:teuthology.orchestra.run.smithi148.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1904.g9eb9 2.9 MB/s | 165 kB 00:00 2020-03-09T00:21:42.176 INFO:teuthology.orchestra.run.smithi050.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 1.2 MB/s | 214 kB 00:00 2020-03-09T00:21:42.283 INFO:teuthology.orchestra.run.smithi148.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 1.1 MB/s | 214 kB 00:00 2020-03-09T00:21:42.312 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-09T00:21:42.369 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:21:42.429 INFO:teuthology.orchestra.run.smithi050.stdout:(6/36): ceph-mon-15.1.0-1904.g9eb9490.el8.x86_6 5.3 MB/s | 4.4 MB 00:00 2020-03-09T00:21:42.505 INFO:teuthology.orchestra.run.smithi050.stdout:(7/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 4.6 MB/s | 1.5 MB 00:00 2020-03-09T00:21:42.531 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-09T00:21:42.587 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-09T00:21:42.621 INFO:teuthology.orchestra.run.smithi050.stdout:(8/36): python3-mako-1.0.6-13.el8.noarch.rpm 818 kB/s | 157 kB 00:00 2020-03-09T00:21:42.634 INFO:teuthology.orchestra.run.smithi050.stdout:(9/36): python3-markupsafe-0.23-19.el8.x86_64.r 301 kB/s | 39 kB 00:00 2020-03-09T00:21:42.641 INFO:teuthology.orchestra.run.smithi148.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 4.2 MB/s | 1.5 MB 00:00 2020-03-09T00:21:42.732 INFO:teuthology.orchestra.run.smithi050.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 276 kB/s | 27 kB 00:00 2020-03-09T00:21:42.738 INFO:teuthology.orchestra.run.smithi050.stdout:(11/36): python3-pytz-2017.2-9.el8.noarch.rpm 463 kB/s | 54 kB 00:00 2020-03-09T00:21:42.812 INFO:teuthology.orchestra.run.smithi148.stdout:(7/36): ceph-mon-15.1.0-1904.g9eb9490.el8.x86_6 3.7 MB/s | 4.4 MB 00:01 2020-03-09T00:21:42.833 INFO:teuthology.orchestra.run.smithi148.stdout:(8/36): python3-mako-1.0.6-13.el8.noarch.rpm 817 kB/s | 157 kB 00:00 2020-03-09T00:21:42.853 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-09T00:21:42.891 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-09T00:21:42.927 INFO:teuthology.orchestra.run.smithi050.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 1.0 MB/s | 195 kB 00:00 2020-03-09T00:21:42.941 INFO:teuthology.orchestra.run.smithi148.stdout:(9/36): python3-markupsafe-0.23-19.el8.x86_64.r 301 kB/s | 39 kB 00:00 2020-03-09T00:21:42.963 INFO:teuthology.orchestra.run.smithi148.stdout:(10/36): python3-pytz-2017.2-9.el8.noarch.rpm 418 kB/s | 54 kB 00:00 2020-03-09T00:21:43.030 INFO:teuthology.orchestra.run.smithi148.stdout:(11/36): python3-webencodings-0.5.1-6.el8.noarc 306 kB/s | 27 kB 00:00 2020-03-09T00:21:43.051 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-09T00:21:43.068 INFO:teuthology.orchestra.run.smithi050.stdout:(13/36): ceph-osd-15.1.0-1904.g9eb9490.el8.x86_ 14 MB/s | 17 MB 00:01 2020-03-09T00:21:43.068 INFO:teuthology.orchestra.run.smithi050.stdout:(14/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 239 kB/s | 34 kB 00:00 2020-03-09T00:21:43.107 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-09T00:21:43.131 INFO:teuthology.orchestra.run.smithi050.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 2.7 MB/s | 176 kB 00:00 2020-03-09T00:21:43.157 INFO:teuthology.orchestra.run.smithi050.stdout:(16/36): python3-werkzeug-0.12.2-4.el8.noarch.r 1.1 MB/s | 457 kB 00:00 2020-03-09T00:21:43.190 INFO:teuthology.orchestra.run.smithi050.stdout:(17/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.4 MB/s | 145 kB 00:00 2020-03-09T00:21:43.198 INFO:teuthology.orchestra.run.smithi050.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.4 MB/s | 11 kB 00:00 2020-03-09T00:21:43.207 INFO:teuthology.orchestra.run.smithi050.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 1.9 MB/s | 18 kB 00:00 2020-03-09T00:21:43.216 INFO:teuthology.orchestra.run.smithi050.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 5.1 MB/s | 49 kB 00:00 2020-03-09T00:21:43.217 INFO:teuthology.orchestra.run.smithi148.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 1.0 MB/s | 195 kB 00:00 2020-03-09T00:21:43.221 INFO:teuthology.orchestra.run.smithi148.stdout:(13/36): python3-werkzeug-0.12.2-4.el8.noarch.r 1.7 MB/s | 457 kB 00:00 2020-03-09T00:21:43.223 INFO:teuthology.orchestra.run.smithi050.stdout:(21/36): python3-requests-2.20.0-2.1.el8_1.noar 793 kB/s | 123 kB 00:00 2020-03-09T00:21:43.225 INFO:teuthology.orchestra.run.smithi050.stdout:(22/36): python3-cherrypy-18.4.0-1.el8.noarch.r 5.5 MB/s | 385 kB 00:00 2020-03-09T00:21:43.227 INFO:teuthology.orchestra.run.smithi050.stdout:(23/36): python3-more-itertools-7.2.0-3.el8.noa 5.5 MB/s | 59 kB 00:00 2020-03-09T00:21:43.237 INFO:teuthology.orchestra.run.smithi050.stdout:(24/36): python3-portend-2.6-1.el8.noarch.rpm 1.4 MB/s | 16 kB 00:00 2020-03-09T00:21:43.237 INFO:teuthology.orchestra.run.smithi050.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 1.9 MB/s | 19 kB 00:00 2020-03-09T00:21:43.245 INFO:teuthology.orchestra.run.smithi050.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.8 MB/s | 24 kB 00:00 2020-03-09T00:21:43.248 INFO:teuthology.orchestra.run.smithi050.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 2.6 MB/s | 30 kB 00:00 2020-03-09T00:21:43.249 INFO:teuthology.orchestra.run.smithi148.stdout:(14/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 1.1 MB/s | 34 kB 00:00 2020-03-09T00:21:43.250 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-09T00:21:43.257 INFO:teuthology.orchestra.run.smithi050.stdout:(28/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.7 MB/s | 23 kB 00:00 2020-03-09T00:21:43.258 INFO:teuthology.orchestra.run.smithi050.stdout:(29/36): python3-trustme-0.5.2-4.el8.noarch.rpm 2.2 MB/s | 27 kB 00:00 2020-03-09T00:21:43.290 INFO:teuthology.orchestra.run.smithi050.stdout:(30/36): python3-pecan-1.3.2-9.el8.noarch.rpm 4.2 MB/s | 283 kB 00:00 2020-03-09T00:21:43.301 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-09T00:21:43.311 INFO:teuthology.orchestra.run.smithi148.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 2.7 MB/s | 176 kB 00:00 2020-03-09T00:21:43.333 INFO:teuthology.orchestra.run.smithi050.stdout:(31/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 578 kB/s | 44 kB 00:00 2020-03-09T00:21:43.336 INFO:teuthology.orchestra.run.smithi050.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.3 MB/s | 185 kB 00:00 2020-03-09T00:21:43.337 INFO:teuthology.orchestra.run.smithi050.stdout:(33/36): python3-cssselect-0.9.2-10.el8.noarch. 862 kB/s | 40 kB 00:00 2020-03-09T00:21:43.368 INFO:teuthology.orchestra.run.smithi050.stdout:(34/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.7 MB/s | 251 kB 00:00 2020-03-09T00:21:43.369 INFO:teuthology.orchestra.run.smithi050.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 2.6 MB/s | 85 kB 00:00 2020-03-09T00:21:43.372 INFO:teuthology.orchestra.run.smithi148.stdout:(16/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.3 MB/s | 145 kB 00:00 2020-03-09T00:21:43.377 INFO:teuthology.orchestra.run.smithi148.stdout:(17/36): python3-requests-2.20.0-2.1.el8_1.noar 794 kB/s | 123 kB 00:00 2020-03-09T00:21:43.393 INFO:teuthology.orchestra.run.smithi050.stdout:(36/36): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2020-03-09T00:21:43.394 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:43.394 INFO:teuthology.orchestra.run.smithi050.stdout:Total 14 MB/s | 29 MB 00:02 2020-03-09T00:21:43.395 INFO:teuthology.orchestra.run.smithi148.stdout:(18/36): python3-cherrypy-18.4.0-1.el8.noarch.r 17 MB/s | 385 kB 00:00 2020-03-09T00:21:43.403 INFO:teuthology.orchestra.run.smithi148.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.0 MB/s | 18 kB 00:00 2020-03-09T00:21:43.414 INFO:teuthology.orchestra.run.smithi148.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 4.8 MB/s | 49 kB 00:00 2020-03-09T00:21:43.415 INFO:teuthology.orchestra.run.smithi148.stdout:(21/36): python3-jaraco-6.2-6.el8.noarch.rpm 301 kB/s | 11 kB 00:00 2020-03-09T00:21:43.417 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:21:43.423 INFO:teuthology.orchestra.run.smithi148.stdout:(22/36): python3-more-itertools-7.2.0-3.el8.noa 6.1 MB/s | 59 kB 00:00 2020-03-09T00:21:43.431 INFO:teuthology.orchestra.run.smithi148.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-03-09T00:21:43.440 INFO:teuthology.orchestra.run.smithi148.stdout:(24/36): python3-simplegeneric-0.8.1-17.el8.noa 2.3 MB/s | 19 kB 00:00 2020-03-09T00:21:43.448 INFO:teuthology.orchestra.run.smithi148.stdout:(25/36): python3-pecan-1.3.2-9.el8.noarch.rpm 8.3 MB/s | 283 kB 00:00 2020-03-09T00:21:43.449 INFO:teuthology.orchestra.run.smithi148.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.6 MB/s | 24 kB 00:00 2020-03-09T00:21:43.456 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-09T00:21:43.457 INFO:teuthology.orchestra.run.smithi148.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.5 MB/s | 30 kB 00:00 2020-03-09T00:21:43.458 INFO:teuthology.orchestra.run.smithi148.stdout:(28/36): python3-trustme-0.5.2-4.el8.noarch.rpm 3.2 MB/s | 27 kB 00:00 2020-03-09T00:21:43.460 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:21:43.460 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:21:43.466 INFO:teuthology.orchestra.run.smithi148.stdout:(29/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.7 MB/s | 23 kB 00:00 2020-03-09T00:21:43.487 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-09T00:21:43.520 INFO:teuthology.orchestra.run.smithi148.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 697 kB/s | 44 kB 00:00 2020-03-09T00:21:43.536 INFO:teuthology.orchestra.run.smithi148.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 2.5 MB/s | 40 kB 00:00 2020-03-09T00:21:43.559 INFO:teuthology.orchestra.run.smithi148.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-09T00:21:43.573 INFO:teuthology.orchestra.run.smithi148.stdout:(33/36): python3-waitress-1.2.1-2.el8.1.noarch. 6.4 MB/s | 241 kB 00:00 2020-03-09T00:21:43.589 INFO:teuthology.orchestra.run.smithi148.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 5.2 MB/s | 85 kB 00:00 2020-03-09T00:21:43.598 INFO:teuthology.orchestra.run.smithi148.stdout:(35/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.3 MB/s | 251 kB 00:00 2020-03-09T00:21:43.640 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-09T00:21:43.690 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-09T00:21:43.794 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:21:43.795 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:21:43.830 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-09T00:21:43.926 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-09T00:21:44.011 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-09T00:21:44.029 INFO:teuthology.orchestra.run.smithi148.stdout:(36/36): ceph-osd-15.1.0-1904.g9eb9490.el8.x86_ 8.2 MB/s | 17 MB 00:02 2020-03-09T00:21:44.032 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:44.033 INFO:teuthology.orchestra.run.smithi148.stdout:Total 11 MB/s | 29 MB 00:02 2020-03-09T00:21:44.052 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:21:44.090 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:21:44.090 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:21:44.172 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-09T00:21:44.208 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-09T00:21:44.402 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-09T00:21:44.404 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-09T00:21:44.424 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:21:44.425 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:21:44.579 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-09T00:21:44.749 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-09T00:21:44.778 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-09T00:21:45.006 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-09T00:21:45.050 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-09T00:21:45.196 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:21:45.262 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-09T00:21:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-09T00:21:45.380 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 31/36 2020-03-09T00:21:45.532 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:45.548 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-09T00:21:45.640 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-09T00:21:45.672 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:21:45.794 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-09T00:21:45.824 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-09T00:21:45.849 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-09T00:21:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-09T00:21:46.074 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-09T00:21:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-09T00:21:46.238 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-09T00:21:46.292 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-09T00:21:46.314 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-09T00:21:46.447 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-09T00:21:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-09T00:21:46.500 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-09T00:21:46.646 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-09T00:21:46.678 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-09T00:21:46.718 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-09T00:21:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-09T00:21:46.836 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-09T00:21:46.927 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:46.985 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-09T00:21:47.019 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-09T00:21:47.076 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:47.105 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-09T00:21:47.124 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-09T00:21:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-09T00:21:47.339 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-09T00:21:47.345 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-09T00:21:47.346 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-09T00:21:47.459 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-09T00:21:47.558 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-09T00:21:47.562 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-09T00:21:47.618 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:47.642 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-09T00:21:47.763 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-09T00:21:47.783 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-09T00:21:47.809 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:47.816 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-09T00:21:47.994 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-09T00:21:48.001 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:48.002 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-09T00:21:48.007 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-09T00:21:48.019 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:48.116 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:48.237 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-09T00:21:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-09T00:21:48.279 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-09T00:21:48.417 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:48.418 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 1/36 2020-03-09T00:21:48.418 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2/36 2020-03-09T00:21:48.418 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 3/36 2020-03-09T00:21:48.418 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 4/36 2020-03-09T00:21:48.418 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 5/36 2020-03-09T00:21:48.419 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 6/36 2020-03-09T00:21:48.419 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-09T00:21:48.419 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-09T00:21:48.419 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-09T00:21:48.420 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-09T00:21:48.420 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-09T00:21:48.420 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-09T00:21:48.420 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-09T00:21:48.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-09T00:21:48.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-09T00:21:48.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-09T00:21:48.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-09T00:21:48.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-09T00:21:48.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-09T00:21:48.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-09T00:21:48.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-09T00:21:48.423 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-09T00:21:48.423 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-09T00:21:48.423 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-09T00:21:48.423 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-09T00:21:48.424 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-09T00:21:48.424 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-09T00:21:48.424 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-09T00:21:48.424 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-09T00:21:48.424 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-09T00:21:48.425 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-09T00:21:48.425 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-09T00:21:48.425 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-09T00:21:48.425 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-09T00:21:48.426 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-09T00:21:48.476 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-09T00:21:48.498 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-09T00:21:48.506 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-09T00:21:48.692 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-09T00:21:48.741 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-09T00:21:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-09T00:21:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-09T00:21:48.975 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-09T00:21:49.090 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-09T00:21:49.114 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-09T00:21:49.190 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-09T00:21:49.190 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:49.190 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:21:49.190 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:49.190 INFO:teuthology.orchestra.run.smithi139.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-09T00:21:49.191 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-09T00:21:49.191 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:49.191 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:49.191 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:49.191 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:49.191 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:21:49.191 INFO:teuthology.orchestra.run.smithi139.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-09T00:21:49.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-09T00:21:49.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-09T00:21:49.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-09T00:21:49.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-09T00:21:49.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-09T00:21:49.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-09T00:21:49.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-09T00:21:49.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-2.6-1.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-09T00:21:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-09T00:21:49.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-09T00:21:49.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-09T00:21:49.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-09T00:21:49.195 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-09T00:21:49.195 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:49.196 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:21:49.222 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-09T00:21:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-09T00:21:49.412 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-09T00:21:49.482 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-09T00:21:49.506 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-09T00:21:49.546 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base 2020-03-09T00:21:49.745 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 31/36 2020-03-09T00:21:49.868 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-09T00:21:49.874 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-09T00:21:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:49.948 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:00 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:50.089 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-09T00:21:50.159 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-09T00:21:50.203 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:21:50.288 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-09T00:21:50.308 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:21:50.309 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:21:50.310 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:21:50.386 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-09T00:21:50.457 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install cephadm 2020-03-09T00:21:50.472 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-09T00:21:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-09T00:21:50.633 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-09T00:21:50.752 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-09T00:21:50.849 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:01 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:50.858 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-09T00:21:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-09T00:21:51.082 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-09T00:21:51.210 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:21:51.211 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:51.211 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2020-03-09T00:21:51.211 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:51.211 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:21:51.211 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 35 k 2020-03-09T00:21:51.211 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:51.212 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:21:51.212 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:51.212 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2020-03-09T00:21:51.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:51.213 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 35 k 2020-03-09T00:21:51.213 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 127 k 2020-03-09T00:21:51.213 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:21:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-09T00:21:51.292 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:51.305 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-09T00:21:51.375 INFO:teuthology.orchestra.run.smithi139.stdout:cephadm-15.1.0-1904.g9eb9490.el8.x86_64.rpm 216 kB/s | 35 kB 00:00 2020-03-09T00:21:51.376 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:51.376 INFO:teuthology.orchestra.run.smithi139.stdout:Total 215 kB/s | 35 kB 00:00 2020-03-09T00:21:51.378 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:21:51.386 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:21:51.386 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:21:51.387 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:21:51.388 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:21:51.473 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:51.480 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-09T00:21:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-09T00:21:51.650 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:21:51.845 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-09T00:21:51.855 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:51.926 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-09T00:21:52.047 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:52.164 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-09T00:21:52.214 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 31/36 2020-03-09T00:21:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:52.358 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:52.414 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:52.432 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 31/36 2020-03-09T00:21:52.529 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 1/36 2020-03-09T00:21:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2/36 2020-03-09T00:21:52.861 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 3/36 2020-03-09T00:21:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 4/36 2020-03-09T00:21:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 5/36 2020-03-09T00:21:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 6/36 2020-03-09T00:21:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-09T00:21:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-09T00:21:52.862 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-09T00:21:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-09T00:21:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-09T00:21:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-09T00:21:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-09T00:21:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-09T00:21:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-09T00:21:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-09T00:21:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-09T00:21:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-09T00:21:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-09T00:21:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-09T00:21:52.864 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-09T00:21:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-09T00:21:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-09T00:21:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-09T00:21:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-09T00:21:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-09T00:21:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-09T00:21:52.865 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-09T00:21:52.866 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-09T00:21:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-09T00:21:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-09T00:21:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-09T00:21:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-09T00:21:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-09T00:21:52.867 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-09T00:21:52.992 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:21:53.236 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:21:53.448 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:21:53.448 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:53.448 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:21:53.448 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:53.449 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:53.449 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:21:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-09T00:21:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:53.562 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:21:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-09T00:21:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-09T00:21:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:21:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-09T00:21:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-09T00:21:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-09T00:21:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-09T00:21:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-09T00:21:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-09T00:21:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-2.6-1.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-09T00:21:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-09T00:21:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-09T00:21:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-09T00:21:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-09T00:21:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-09T00:21:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-09T00:21:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-09T00:21:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:53.568 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:21:53.584 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-immutable-object-cache 2020-03-09T00:21:53.753 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:53.890 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2020-03-09T00:21:53.936 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:53.985 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:04 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:54.164 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 32/36 2020-03-09T00:21:54.292 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:05 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:54.320 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:54.329 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:54.392 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:21:54.393 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:54.393 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2020-03-09T00:21:54.393 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:54.394 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:21:54.394 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 203 k 2020-03-09T00:21:54.394 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:54.394 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:21:54.394 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:54.394 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2020-03-09T00:21:54.394 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:54.395 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 203 k 2020-03-09T00:21:54.395 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 726 k 2020-03-09T00:21:54.395 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:21:54.453 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:54.566 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-immutable-object-cache-15.1.0-1904.g9eb949 1.2 MB/s | 203 kB 00:00 2020-03-09T00:21:54.567 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:54.567 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.1 MB/s | 203 kB 00:00 2020-03-09T00:21:54.569 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:21:54.571 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:21:54.581 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:21:54.581 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:21:54.627 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:21:54.627 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:21:54.644 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:54.677 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:21:54.678 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:21:54.678 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:21:54.737 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 33/36 2020-03-09T00:21:54.776 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:21:54.790 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:54.808 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:54.829 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2020-03-09T00:21:54.871 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:54.917 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:54.942 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:21:55.074 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 34/36 2020-03-09T00:21:55.219 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:06 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:55.231 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:55.231 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 1/36 2020-03-09T00:21:55.232 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2/36 2020-03-09T00:21:55.232 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 3/36 2020-03-09T00:21:55.232 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 4/36 2020-03-09T00:21:55.232 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 5/36 2020-03-09T00:21:55.232 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 6/36 2020-03-09T00:21:55.232 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-09T00:21:55.232 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-09T00:21:55.233 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-09T00:21:55.234 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-09T00:21:55.235 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-09T00:21:55.235 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-09T00:21:55.235 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-09T00:21:55.235 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-09T00:21:55.235 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-09T00:21:55.235 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-09T00:21:55.235 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-09T00:21:55.236 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-09T00:21:55.236 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-09T00:21:55.236 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-09T00:21:55.236 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-09T00:21:55.239 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:21:55.437 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:55.452 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:21:55.452 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:55.453 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:21:55.453 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:55.453 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:55.453 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:21:55.458 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 35/36 2020-03-09T00:21:55.585 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:21:55.586 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:55.586 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2020-03-09T00:21:55.586 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:55.586 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:21:55.586 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 35 k 2020-03-09T00:21:55.586 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:55.587 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:21:55.587 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:55.587 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2020-03-09T00:21:55.587 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:55.587 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 35 k 2020-03-09T00:21:55.587 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 127 k 2020-03-09T00:21:55.587 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:21:55.602 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:55.697 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-15.1.0-1904.g9eb9490.el8.x86_64.rpm 321 kB/s | 35 kB 00:00 2020-03-09T00:21:55.697 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:55.697 INFO:teuthology.orchestra.run.smithi104.stdout:Total 317 kB/s | 35 kB 00:00 2020-03-09T00:21:55.699 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:21:55.710 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:21:55.710 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:21:55.711 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:21:55.711 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:21:55.753 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:21:55.772 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr 2020-03-09T00:21:55.918 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 36/36 2020-03-09T00:21:55.918 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 1/36 2020-03-09T00:21:55.918 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2/36 2020-03-09T00:21:55.918 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 3/36 2020-03-09T00:21:55.919 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 4/36 2020-03-09T00:21:55.919 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 5/36 2020-03-09T00:21:55.919 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.n 6/36 2020-03-09T00:21:55.919 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-09T00:21:55.919 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-09T00:21:55.919 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-09T00:21:55.919 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-09T00:21:55.920 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-09T00:21:55.920 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-09T00:21:55.920 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-09T00:21:55.920 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-09T00:21:55.920 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-09T00:21:55.921 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-09T00:21:55.921 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-09T00:21:55.922 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-09T00:21:55.923 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-09T00:21:55.923 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-09T00:21:55.923 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-09T00:21:55.923 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-09T00:21:55.923 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-09T00:21:55.923 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-09T00:21:55.923 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-09T00:21:55.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-09T00:21:55.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-09T00:21:55.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-09T00:21:55.998 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-09T00:21:55.999 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:55.999 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:21:55.999 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:55.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-09T00:21:55.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-09T00:21:55.999 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:55.999 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-09T00:21:56.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-09T00:21:56.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-09T00:21:56.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-09T00:21:56.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-09T00:21:56.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-09T00:21:56.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-09T00:21:56.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-09T00:21:56.002 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-09T00:21:56.002 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-09T00:21:56.002 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-09T00:21:56.002 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-09T00:21:56.002 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-09T00:21:56.002 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-09T00:21:56.002 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-2.6-1.el8.noarch 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-09T00:21:56.003 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-09T00:21:56.004 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-09T00:21:56.004 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-09T00:21:56.004 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:56.004 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:21:56.172 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:07 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:56.355 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base 2020-03-09T00:21:56.444 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:21:56.553 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:21:56.555 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:21:56.555 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:21:56.685 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-dashboard 2020-03-09T00:21:56.758 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:07 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:56.833 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-09T00:21:56.833 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:56.833 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:21:56.833 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.834 INFO:teuthology.orchestra.run.smithi050.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-09T00:21:56.834 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-09T00:21:56.834 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.834 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.834 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.834 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:56.834 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:21:56.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-09T00:21:56.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-09T00:21:56.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-09T00:21:56.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-09T00:21:56.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-09T00:21:56.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-09T00:21:56.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-09T00:21:56.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-09T00:21:56.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-09T00:21:56.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-09T00:21:56.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-09T00:21:56.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-09T00:21:56.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-09T00:21:56.837 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-09T00:21:56.837 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-09T00:21:56.837 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-09T00:21:56.837 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-09T00:21:56.837 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-2.6-1.el8.noarch 2020-03-09T00:21:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-09T00:21:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-09T00:21:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-09T00:21:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-09T00:21:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-09T00:21:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-09T00:21:56.838 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-09T00:21:56.839 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-09T00:21:56.839 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-09T00:21:56.839 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-09T00:21:56.839 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:56.839 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:21:57.023 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:21:57.078 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:08 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:57.105 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:21:57.127 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:21:57.129 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:21:57.129 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:21:57.145 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2020-03-09T00:21:57.258 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install cephadm 2020-03-09T00:21:57.372 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:21:57.453 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:21:57.453 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:57.453 INFO:teuthology.orchestra.run.smithi139.stdout: Package ArchVersion Repository Size 2020-03-09T00:21:57.454 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:57.454 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:21:57.454 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 3.9 M 2020-03-09T00:21:57.454 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:21:57.454 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 22 k 2020-03-09T00:21:57.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt noarch1.6.1-2.el8 BaseOS 43 k 2020-03-09T00:21:57.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-09T00:21:57.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-09T00:21:57.455 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:57.455 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:21:57.455 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:21:57.455 INFO:teuthology.orchestra.run.smithi139.stdout:Install 5 Packages 2020-03-09T00:21:57.455 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:21:57.455 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 4.2 M 2020-03-09T00:21:57.456 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 19 M 2020-03-09T00:21:57.456 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:21:57.575 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:08 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:21:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:57.585 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:21:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:57.585 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:21:57.651 INFO:teuthology.orchestra.run.smithi139.stdout:(1/5): ceph-grafana-dashboards-15.1.0-1904.g9eb 205 kB/s | 22 kB 00:00 2020-03-09T00:21:57.651 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:08 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:57.677 INFO:teuthology.orchestra.run.smithi139.stdout:(2/5): python3-jwt-1.6.1-2.el8.noarch.rpm 314 kB/s | 43 kB 00:00 2020-03-09T00:21:57.698 INFO:teuthology.orchestra.run.smithi139.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 730 kB/s | 34 kB 00:00 2020-03-09T00:21:57.739 INFO:teuthology.orchestra.run.smithi139.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 3.1 MB/s | 197 kB 00:00 2020-03-09T00:21:57.754 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2020-03-09T00:21:57.852 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:21:57.861 INFO:teuthology.orchestra.run.smithi139.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1904.g9eb9490. 12 MB/s | 3.9 MB 00:00 2020-03-09T00:21:57.863 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:57.864 INFO:teuthology.orchestra.run.smithi139.stdout:Total 10 MB/s | 4.2 MB 00:00 2020-03-09T00:21:57.868 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:21:57.877 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:21:57.877 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:21:57.963 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:21:57.964 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:21:57.964 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:21:58.002 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:21:58.003 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:21:58.023 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:21:58.024 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:58.024 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2020-03-09T00:21:58.025 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:58.025 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:21:58.025 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 35 k 2020-03-09T00:21:58.025 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:58.025 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:21:58.025 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:21:58.026 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2020-03-09T00:21:58.026 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:21:58.026 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 35 k 2020-03-09T00:21:58.026 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 127 k 2020-03-09T00:21:58.027 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:21:58.118 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2020-03-09T00:21:58.136 INFO:teuthology.orchestra.run.smithi148.stdout:cephadm-15.1.0-1904.g9eb9490.el8.x86_64.rpm 319 kB/s | 35 kB 00:00 2020-03-09T00:21:58.136 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:58.137 INFO:teuthology.orchestra.run.smithi148.stdout:Total 315 kB/s | 35 kB 00:00 2020-03-09T00:21:58.139 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:21:58.146 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:21:58.146 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:21:58.148 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:21:58.148 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:21:58.156 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:09 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:58.187 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:21:58.244 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:21:58.439 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-09T00:21:58.546 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:09 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:21:58.601 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:21:58.602 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:58.602 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2020-03-09T00:21:58.603 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:58.603 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:21:58.603 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 203 k 2020-03-09T00:21:58.603 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:58.603 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:21:58.604 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:21:58.604 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2020-03-09T00:21:58.604 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:58.604 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 203 k 2020-03-09T00:21:58.604 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 726 k 2020-03-09T00:21:58.605 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:21:58.650 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-09T00:21:58.856 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:21:58.948 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:21:58.949 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:58.949 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2020-03-09T00:21:58.949 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:58.950 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:21:58.950 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 35 k 2020-03-09T00:21:58.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:58.950 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:21:58.950 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:21:58.950 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2020-03-09T00:21:58.950 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:21:58.951 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 35 k 2020-03-09T00:21:58.951 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 127 k 2020-03-09T00:21:58.951 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:21:58.956 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-15.1.0-1904.g9eb949 575 kB/s | 203 kB 00:00 2020-03-09T00:21:58.956 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:58.957 INFO:teuthology.orchestra.run.smithi104.stdout:Total 573 kB/s | 203 kB 00:00 2020-03-09T00:21:58.959 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:21:58.971 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:21:58.971 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:21:59.018 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:21:59.019 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:21:59.075 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-15.1.0-1904.g9eb9490.el8.x86_64.rpm 282 kB/s | 35 kB 00:00 2020-03-09T00:21:59.076 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:21:59.076 INFO:teuthology.orchestra.run.smithi050.stdout:Total 279 kB/s | 35 kB 00:00 2020-03-09T00:21:59.078 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:21:59.086 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:21:59.086 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:21:59.088 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:21:59.088 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:21:59.132 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:21:59.136 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:21:59.312 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:21:59.553 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:21:59.644 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:21:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:21:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:59.882 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:21:59.882 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:21:59.883 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:21:59.883 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:21:59.886 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 4/5 2020-03-09T00:22:00.165 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:00.198 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2020-03-09T00:22:00.398 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:00.398 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 1/5 2020-03-09T00:22:00.398 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2/5 2020-03-09T00:22:00.398 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:22:00.398 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-09T00:22:00.417 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:00.598 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:11 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:00.665 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:00.665 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:00.665 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:00.665 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:00.666 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:00.666 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:00.713 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-09T00:22:00.714 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:00.714 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:00.714 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:00.714 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:00.714 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-09T00:22:00.714 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-09T00:22:00.714 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-09T00:22:00.715 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:00.715 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:00.795 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-immutable-object-cache 2020-03-09T00:22:00.863 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-09T00:22:00.896 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:00.936 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:01.002 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:01.003 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:01.003 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:01.176 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2020-03-09T00:22:01.182 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:01.200 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:12 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:01.254 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:12 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:01.394 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:01.394 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:01.395 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:01.395 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:01.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:01.395 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:01.554 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2020-03-09T00:22:01.572 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:12 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:01.574 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:01.575 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:01.575 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2020-03-09T00:22:01.575 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:01.575 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:01.575 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 203 k 2020-03-09T00:22:01.575 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:01.575 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:01.576 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:01.576 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2020-03-09T00:22:01.576 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:01.576 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 203 k 2020-03-09T00:22:01.577 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 726 k 2020-03-09T00:22:01.577 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:01.616 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:01.617 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:01.617 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2020-03-09T00:22:01.617 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:01.618 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local 2020-03-09T00:22:01.618 INFO:teuthology.orchestra.run.smithi139.stdout: noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 8.6 M 2020-03-09T00:22:01.621 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:22:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.3-2.el8 AppStream 4.3 M 2020-03-09T00:22:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads x86_64 0.3.3-2.el8 AppStream 4.4 M 2020-03-09T00:22:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.14.3-9.el8 AppStream 3.7 M 2020-03-09T00:22:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py 2020-03-09T00:22:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: x86_64 1:1.14.3-9.el8 AppStream 225 k 2020-03-09T00:22:01.621 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.0.0-20.module_el8.1.0+245+c39af44f AppStream 15 M 2020-03-09T00:22:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 8.3.1-4.5.el8 BaseOS 638 k 2020-03-09T00:22:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 8.3.1-4.5.el8 BaseOS 169 k 2020-03-09T00:22:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:01.622 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:01.622 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:01.622 INFO:teuthology.orchestra.run.smithi139.stdout:Install 8 Packages 2020-03-09T00:22:01.622 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:01.623 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 37 M 2020-03-09T00:22:01.623 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 205 M 2020-03-09T00:22:01.623 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:01.747 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-immutable-object-cache-15.1.0-1904.g9eb949 1.2 MB/s | 203 kB 00:00 2020-03-09T00:22:01.747 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:01.748 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.2 MB/s | 203 kB 00:00 2020-03-09T00:22:01.750 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:01.762 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:01.762 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:01.808 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:01.808 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:01.934 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:01.935 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:01.935 INFO:teuthology.orchestra.run.smithi104.stdout: Package ArchVersion Repository Size 2020-03-09T00:22:01.935 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:01.935 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:01.935 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 3.9 M 2020-03-09T00:22:01.935 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:22:01.935 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 22 k 2020-03-09T00:22:01.936 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch1.6.1-2.el8 BaseOS 43 k 2020-03-09T00:22:01.936 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-09T00:22:01.936 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-09T00:22:01.936 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:01.936 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:01.936 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:01.936 INFO:teuthology.orchestra.run.smithi104.stdout:Install 5 Packages 2020-03-09T00:22:01.937 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:01.937 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 4.2 M 2020-03-09T00:22:01.937 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 19 M 2020-03-09T00:22:01.937 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:01.955 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:01.987 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:12 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:02.110 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:22:02.249 INFO:teuthology.orchestra.run.smithi104.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 342 kB/s | 43 kB 00:00 2020-03-09T00:22:02.296 INFO:teuthology.orchestra.run.smithi104.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 726 kB/s | 34 kB 00:00 2020-03-09T00:22:02.324 INFO:teuthology.orchestra.run.smithi104.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 6.9 MB/s | 197 kB 00:00 2020-03-09T00:22:02.355 INFO:teuthology.orchestra.run.smithi104.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1904.g9eb 97 kB/s | 22 kB 00:00 2020-03-09T00:22:02.358 INFO:teuthology.orchestra.run.smithi139.stdout:(1/8): ceph-mgr-diskprediction-local-15.1.0-190 17 MB/s | 8.6 MB 00:00 2020-03-09T00:22:02.386 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:02.387 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:02.387 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2020-03-09T00:22:02.387 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:02.387 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:02.387 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 203 k 2020-03-09T00:22:02.387 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:02.387 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:02.388 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:02.388 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2020-03-09T00:22:02.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:02.388 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 203 k 2020-03-09T00:22:02.389 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 726 k 2020-03-09T00:22:02.389 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:02.434 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:22:02.458 INFO:teuthology.orchestra.run.smithi139.stdout:(2/8): openblas-0.3.3-2.el8.x86_64.rpm 7.3 MB/s | 4.3 MB 00:00 2020-03-09T00:22:02.488 INFO:teuthology.orchestra.run.smithi139.stdout:(3/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2020-03-09T00:22:02.588 INFO:teuthology.orchestra.run.smithi104.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1904.g9eb9490. 8.4 MB/s | 3.9 MB 00:00 2020-03-09T00:22:02.590 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:02.591 INFO:teuthology.orchestra.run.smithi104.stdout:Total 6.4 MB/s | 4.2 MB 00:00 2020-03-09T00:22:02.595 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:02.604 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:02.604 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:02.640 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-15.1.0-1904.g9eb949 803 kB/s | 203 kB 00:00 2020-03-09T00:22:02.641 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:02.641 INFO:teuthology.orchestra.run.smithi050.stdout:Total 800 kB/s | 203 kB 00:00 2020-03-09T00:22:02.644 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:02.657 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:02.657 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:02.678 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:22:02.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:02.679 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:02.679 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:02.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:02.679 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:02.693 INFO:teuthology.orchestra.run.smithi139.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 958 kB/s | 225 kB 00:00 2020-03-09T00:22:02.707 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:02.707 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:02.739 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:02.739 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:02.874 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:02.923 INFO:teuthology.orchestra.run.smithi139.stdout:(5/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 6.6 MB/s | 3.7 MB 00:00 2020-03-09T00:22:02.981 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:02.997 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr 2020-03-09T00:22:03.003 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:22:03.009 INFO:teuthology.orchestra.run.smithi139.stdout:(6/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 2.0 MB/s | 638 kB 00:00 2020-03-09T00:22:03.127 INFO:teuthology.orchestra.run.smithi139.stdout:(7/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 830 kB/s | 169 kB 00:00 2020-03-09T00:22:03.209 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-09T00:22:03.360 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:22:03.403 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:14 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:03.570 INFO:teuthology.orchestra.run.smithi139.stdout:(8/8): python3-scipy-1.0.0-20.module_el8.1.0+24 13 MB/s | 15 MB 00:01 2020-03-09T00:22:03.571 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:03.571 INFO:teuthology.orchestra.run.smithi139.stdout:Total 19 MB/s | 37 MB 00:01 2020-03-09T00:22:03.583 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:03.604 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:03.604 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:03.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.e 1/1 2020-03-09T00:22:03.624 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:03.624 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:03.625 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:03.625 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:03.625 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:03.702 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:03.794 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-09T00:22:03.807 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:03.808 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:03.809 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:03.919 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:03.919 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:03.920 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2020-03-09T00:22:03.933 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-dashboard 2020-03-09T00:22:04.327 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:15 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:04.346 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:15 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:04.651 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:04.677 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:04.709 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:04.710 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:04.710 INFO:teuthology.orchestra.run.smithi148.stdout: Package ArchVersion Repository Size 2020-03-09T00:22:04.710 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:04.710 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:04.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 3.9 M 2020-03-09T00:22:04.710 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:22:04.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 22 k 2020-03-09T00:22:04.711 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt noarch1.6.1-2.el8 BaseOS 43 k 2020-03-09T00:22:04.711 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-09T00:22:04.711 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-09T00:22:04.711 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:04.711 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:04.711 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:04.712 INFO:teuthology.orchestra.run.smithi148.stdout:Install 5 Packages 2020-03-09T00:22:04.712 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:04.712 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 4.2 M 2020-03-09T00:22:04.712 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 19 M 2020-03-09T00:22:04.713 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:04.765 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:04.766 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:04.805 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:04.915 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2020-03-09T00:22:04.950 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:05.009 INFO:teuthology.orchestra.run.smithi148.stdout:(1/5): ceph-grafana-dashboards-15.1.0-1904.g9eb 202 kB/s | 22 kB 00:00 2020-03-09T00:22:05.024 INFO:teuthology.orchestra.run.smithi148.stdout:(2/5): python3-jwt-1.6.1-2.el8.noarch.rpm 341 kB/s | 43 kB 00:00 2020-03-09T00:22:05.059 INFO:teuthology.orchestra.run.smithi148.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 692 kB/s | 34 kB 00:00 2020-03-09T00:22:05.061 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:22:05.093 INFO:teuthology.orchestra.run.smithi148.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 2.8 MB/s | 197 kB 00:00 2020-03-09T00:22:05.098 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:05.223 INFO:teuthology.orchestra.run.smithi148.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1904.g9eb9490. 12 MB/s | 3.9 MB 00:00 2020-03-09T00:22:05.225 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:05.226 INFO:teuthology.orchestra.run.smithi148.stdout:Total 8.1 MB/s | 4.2 MB 00:00 2020-03-09T00:22:05.230 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:05.239 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:05.239 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:05.343 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:16 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:05.372 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:05.373 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:05.609 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:05.632 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:05.763 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:05.764 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: Package ArchVersion Repository Size 2020-03-09T00:22:05.764 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:05.764 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 3.9 M 2020-03-09T00:22:05.765 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:22:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 22 k 2020-03-09T00:22:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt noarch1.6.1-2.el8 BaseOS 43 k 2020-03-09T00:22:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-09T00:22:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-09T00:22:05.765 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:05.766 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:05.766 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:05.766 INFO:teuthology.orchestra.run.smithi050.stdout:Install 5 Packages 2020-03-09T00:22:05.766 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:05.767 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 4.2 M 2020-03-09T00:22:05.767 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 19 M 2020-03-09T00:22:05.767 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:05.789 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:05.806 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-09T00:22:05.993 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-09T00:22:06.069 INFO:teuthology.orchestra.run.smithi050.stdout:(1/5): ceph-grafana-dashboards-15.1.0-1904.g9eb 172 kB/s | 22 kB 00:00 2020-03-09T00:22:06.070 INFO:teuthology.orchestra.run.smithi050.stdout:(2/5): python3-jwt-1.6.1-2.el8.noarch.rpm 326 kB/s | 43 kB 00:00 2020-03-09T00:22:06.107 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 4/5 2020-03-09T00:22:06.161 INFO:teuthology.orchestra.run.smithi050.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 375 kB/s | 34 kB 00:00 2020-03-09T00:22:06.191 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:22:06.191 INFO:teuthology.orchestra.run.smithi050.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 197 kB 00:00 2020-03-09T00:22:06.285 INFO:teuthology.orchestra.run.smithi050.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1904.g9eb9490. 11 MB/s | 3.9 MB 00:00 2020-03-09T00:22:06.287 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:06.287 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.0 MB/s | 4.2 MB 00:00 2020-03-09T00:22:06.293 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:06.303 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:06.304 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:06.330 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:06.402 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:06.489 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:06.495 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:06.495 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:06.611 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:06.612 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 1/5 2020-03-09T00:22:06.612 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2/5 2020-03-09T00:22:06.612 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:22:06.612 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-09T00:22:06.891 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-09T00:22:06.891 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:06.891 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:06.892 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:06.892 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:06.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-09T00:22:06.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-09T00:22:06.892 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-09T00:22:06.892 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:06.892 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:07.092 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-09T00:22:07.167 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:07.186 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 4/5 2020-03-09T00:22:07.188 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-09T00:22:07.407 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-09T00:22:07.467 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:07.486 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:18 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:07.535 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:07.636 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:22:07.668 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:07.668 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 1/5 2020-03-09T00:22:07.668 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2/5 2020-03-09T00:22:07.668 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:22:07.668 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-09T00:22:07.755 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:07.849 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:07.850 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:07.850 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2020-03-09T00:22:07.851 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:07.851 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local 2020-03-09T00:22:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 8.6 M 2020-03-09T00:22:07.851 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:22:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.3-2.el8 AppStream 4.3 M 2020-03-09T00:22:07.851 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads x86_64 0.3.3-2.el8 AppStream 4.4 M 2020-03-09T00:22:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.14.3-9.el8 AppStream 3.7 M 2020-03-09T00:22:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py 2020-03-09T00:22:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: x86_64 1:1.14.3-9.el8 AppStream 225 k 2020-03-09T00:22:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.0.0-20.module_el8.1.0+245+c39af44f AppStream 15 M 2020-03-09T00:22:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 8.3.1-4.5.el8 BaseOS 638 k 2020-03-09T00:22:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 8.3.1-4.5.el8 BaseOS 169 k 2020-03-09T00:22:07.852 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:07.853 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:07.853 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:07.853 INFO:teuthology.orchestra.run.smithi104.stdout:Install 8 Packages 2020-03-09T00:22:07.853 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:07.854 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 37 M 2020-03-09T00:22:07.854 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 205 M 2020-03-09T00:22:07.854 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:07.911 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-09T00:22:07.912 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:07.912 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:07.912 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:07.912 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:07.912 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-09T00:22:07.912 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-09T00:22:07.912 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-09T00:22:07.913 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:07.913 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:08.078 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-09T00:22:08.474 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:19 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:08.704 INFO:teuthology.orchestra.run.smithi104.stdout:(1/8): ceph-mgr-diskprediction-local-15.1.0-190 17 MB/s | 8.6 MB 00:00 2020-03-09T00:22:08.750 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 4/5 2020-03-09T00:22:08.755 INFO:teuthology.orchestra.run.smithi104.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 8.1 MB/s | 4.4 MB 00:00 2020-03-09T00:22:08.779 INFO:teuthology.orchestra.run.smithi104.stdout:(3/8): openblas-0.3.3-2.el8.x86_64.rpm 7.6 MB/s | 4.3 MB 00:00 2020-03-09T00:22:08.849 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:08.851 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:08.851 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2020-03-09T00:22:08.851 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:08.851 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:08.851 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local 2020-03-09T00:22:08.851 INFO:teuthology.orchestra.run.smithi148.stdout: noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 8.6 M 2020-03-09T00:22:08.851 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:22:08.852 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.3-2.el8 AppStream 4.3 M 2020-03-09T00:22:08.852 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads x86_64 0.3.3-2.el8 AppStream 4.4 M 2020-03-09T00:22:08.852 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.14.3-9.el8 AppStream 3.7 M 2020-03-09T00:22:08.852 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py 2020-03-09T00:22:08.852 INFO:teuthology.orchestra.run.smithi148.stdout: x86_64 1:1.14.3-9.el8 AppStream 225 k 2020-03-09T00:22:08.852 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.0.0-20.module_el8.1.0+245+c39af44f AppStream 15 M 2020-03-09T00:22:08.852 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 8.3.1-4.5.el8 BaseOS 638 k 2020-03-09T00:22:08.853 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 8.3.1-4.5.el8 BaseOS 169 k 2020-03-09T00:22:08.853 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:08.853 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:08.853 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:08.853 INFO:teuthology.orchestra.run.smithi148.stdout:Install 8 Packages 2020-03-09T00:22:08.853 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:08.854 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 37 M 2020-03-09T00:22:08.854 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 205 M 2020-03-09T00:22:08.854 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:08.981 INFO:teuthology.orchestra.run.smithi104.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 997 kB/s | 225 kB 00:00 2020-03-09T00:22:09.035 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:09.250 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 5/5 2020-03-09T00:22:09.250 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.n 1/5 2020-03-09T00:22:09.250 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2/5 2020-03-09T00:22:09.251 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-09T00:22:09.251 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-09T00:22:09.263 INFO:teuthology.orchestra.run.smithi104.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 2.2 MB/s | 638 kB 00:00 2020-03-09T00:22:09.303 INFO:teuthology.orchestra.run.smithi104.stdout:(6/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 6.2 MB/s | 3.7 MB 00:00 2020-03-09T00:22:09.335 INFO:teuthology.orchestra.run.smithi104.stdout:(7/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 2.3 MB/s | 169 kB 00:00 2020-03-09T00:22:09.502 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-09T00:22:09.502 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:09.502 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:09.503 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:09.503 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:09.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-09T00:22:09.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-09T00:22:09.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-09T00:22:09.503 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:09.503 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:09.645 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-09T00:22:09.705 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:09.733 INFO:teuthology.orchestra.run.smithi148.stdout:(1/8): ceph-mgr-diskprediction-local-15.1.0-190 17 MB/s | 8.6 MB 00:00 2020-03-09T00:22:09.776 INFO:teuthology.orchestra.run.smithi148.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 8.2 MB/s | 4.4 MB 00:00 2020-03-09T00:22:09.788 INFO:teuthology.orchestra.run.smithi148.stdout:(3/8): openblas-0.3.3-2.el8.x86_64.rpm 7.8 MB/s | 4.3 MB 00:00 2020-03-09T00:22:09.790 INFO:teuthology.orchestra.run.smithi104.stdout:(8/8): python3-scipy-1.0.0-20.module_el8.1.0+24 14 MB/s | 15 MB 00:01 2020-03-09T00:22:09.791 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:09.791 INFO:teuthology.orchestra.run.smithi104.stdout:Total 19 MB/s | 37 MB 00:01 2020-03-09T00:22:09.802 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:09.821 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:09.822 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:09.980 INFO:teuthology.orchestra.run.smithi148.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 1.1 MB/s | 225 kB 00:00 2020-03-09T00:22:10.070 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:21 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:10.160 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:10.161 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:10.304 INFO:teuthology.orchestra.run.smithi148.stdout:(5/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 6.6 MB/s | 3.7 MB 00:00 2020-03-09T00:22:10.326 INFO:teuthology.orchestra.run.smithi148.stdout:(6/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 1.8 MB/s | 638 kB 00:00 2020-03-09T00:22:10.461 INFO:teuthology.orchestra.run.smithi148.stdout:(7/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 1.1 MB/s | 169 kB 00:00 2020-03-09T00:22:10.472 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:10.474 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:10.474 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2020-03-09T00:22:10.474 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:10.474 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local 2020-03-09T00:22:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 8.6 M 2020-03-09T00:22:10.475 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:22:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.3-2.el8 AppStream 4.3 M 2020-03-09T00:22:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads x86_64 0.3.3-2.el8 AppStream 4.4 M 2020-03-09T00:22:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.14.3-9.el8 AppStream 3.7 M 2020-03-09T00:22:10.475 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py 2020-03-09T00:22:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 1:1.14.3-9.el8 AppStream 225 k 2020-03-09T00:22:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.0.0-20.module_el8.1.0+245+c39af44f AppStream 15 M 2020-03-09T00:22:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 8.3.1-4.5.el8 BaseOS 638 k 2020-03-09T00:22:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 8.3.1-4.5.el8 BaseOS 169 k 2020-03-09T00:22:10.476 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:10.476 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:10.476 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:10.477 INFO:teuthology.orchestra.run.smithi050.stdout:Install 8 Packages 2020-03-09T00:22:10.477 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:10.477 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 37 M 2020-03-09T00:22:10.477 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 205 M 2020-03-09T00:22:10.478 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:10.856 INFO:teuthology.orchestra.run.smithi148.stdout:(8/8): python3-scipy-1.0.0-20.module_el8.1.0+24 14 MB/s | 15 MB 00:01 2020-03-09T00:22:10.857 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:10.858 INFO:teuthology.orchestra.run.smithi148.stdout:Total 18 MB/s | 37 MB 00:02 2020-03-09T00:22:10.870 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:10.882 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:10.891 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:10.891 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:10.989 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-09T00:22:11.021 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:11.155 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:11.160 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:11.276 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:11.276 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:11.351 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:11.395 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:11.395 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 1/8 2020-03-09T00:22:11.395 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-09T00:22:11.395 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:11.395 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-09T00:22:11.395 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:11.396 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-09T00:22:11.396 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-09T00:22:11.687 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-09T00:22:11.687 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:11.687 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:11.687 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:11.687 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-09T00:22:11.687 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-09T00:22:11.687 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:11.688 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:11.688 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-09T00:22:11.688 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:11.688 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:11.688 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:11.688 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:11.819 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-rook 2020-03-09T00:22:11.881 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:12.034 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:12.216 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:23 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:12.322 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:12.475 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:12.570 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:12.572 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:12.574 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:12.574 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2020-03-09T00:22:12.574 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:12.574 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 24 k 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-2.el8 AppStream 27 k 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 1.10-11.el8 AppStream 20 k 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.3.7-6.el8 AppStream 126 k 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 AppStream 110 k 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 2.1.0-1.el8 BaseOS 155 k 2020-03-09T00:22:12.575 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 BaseOS 43 k 2020-03-09T00:22:12.576 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-09T00:22:12.576 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-09T00:22:12.576 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-09T00:22:12.576 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-09T00:22:12.576 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-09T00:22:12.577 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-09T00:22:12.577 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:12.577 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:12.577 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:12.577 INFO:teuthology.orchestra.run.smithi139.stdout:Install 13 Packages 2020-03-09T00:22:12.578 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:12.579 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 1.9 M 2020-03-09T00:22:12.579 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 23 M 2020-03-09T00:22:12.579 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:12.613 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:12.684 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:12.766 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:13.070 INFO:teuthology.orchestra.run.smithi139.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 301 kB/s | 27 kB 00:00 2020-03-09T00:22:13.070 INFO:teuthology.orchestra.run.smithi139.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 225 kB/s | 20 kB 00:00 2020-03-09T00:22:13.091 INFO:teuthology.orchestra.run.smithi139.stdout:(3/13): ceph-mgr-rook-15.1.0-1904.g9eb9490.el8. 221 kB/s | 24 kB 00:00 2020-03-09T00:22:13.263 INFO:teuthology.orchestra.run.smithi139.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 904 kB/s | 155 kB 00:00 2020-03-09T00:22:13.269 INFO:teuthology.orchestra.run.smithi139.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 632 kB/s | 126 kB 00:00 2020-03-09T00:22:13.271 INFO:teuthology.orchestra.run.smithi139.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 546 kB/s | 110 kB 00:00 2020-03-09T00:22:13.298 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:13.303 INFO:teuthology.orchestra.run.smithi139.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2020-03-09T00:22:13.327 INFO:teuthology.orchestra.run.smithi139.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 574 kB/s | 33 kB 00:00 2020-03-09T00:22:13.327 INFO:teuthology.orchestra.run.smithi139.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 291 kB/s | 16 kB 00:00 2020-03-09T00:22:13.365 INFO:teuthology.orchestra.run.smithi139.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 1.9 MB/s | 73 kB 00:00 2020-03-09T00:22:13.368 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:13.378 INFO:teuthology.orchestra.run.smithi139.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 1.1 MB/s | 81 kB 00:00 2020-03-09T00:22:13.384 INFO:teuthology.orchestra.run.smithi139.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 3.1 MB/s | 61 kB 00:00 2020-03-09T00:22:13.433 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:13.445 INFO:teuthology.orchestra.run.smithi139.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 9.5 MB/s | 1.1 MB 00:00 2020-03-09T00:22:13.445 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:13.446 INFO:teuthology.orchestra.run.smithi139.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2020-03-09T00:22:13.452 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:13.466 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:13.466 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:13.538 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:13.538 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:13.696 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:13.909 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:13.972 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:14.107 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:14.147 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:14.676 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-09T00:22:14.811 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:14.903 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-09T00:22:15.094 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:15.116 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-09T00:22:15.312 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-09T00:22:15.334 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:15.520 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-09T00:22:15.541 INFO:teuthology.orchestra.run.smithi050.stdout:(1/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2020-03-09T00:22:15.580 INFO:teuthology.orchestra.run.smithi050.stdout:(2/8): ceph-mgr-diskprediction-local-15.1.0-190 17 MB/s | 8.6 MB 00:00 2020-03-09T00:22:15.681 INFO:teuthology.orchestra.run.smithi050.stdout:(3/8): openblas-0.3.3-2.el8.x86_64.rpm 7.2 MB/s | 4.3 MB 00:00 2020-03-09T00:22:15.761 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-09T00:22:15.774 INFO:teuthology.orchestra.run.smithi050.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 1.1 MB/s | 225 kB 00:00 2020-03-09T00:22:15.848 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:16.009 INFO:teuthology.orchestra.run.smithi050.stdout:(5/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 8.0 MB/s | 3.7 MB 00:00 2020-03-09T00:22:16.020 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-09T00:22:16.056 INFO:teuthology.orchestra.run.smithi050.stdout:(6/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 2.2 MB/s | 638 kB 00:00 2020-03-09T00:22:16.165 INFO:teuthology.orchestra.run.smithi050.stdout:(7/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 1.1 MB/s | 169 kB 00:00 2020-03-09T00:22:16.234 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-09T00:22:16.657 INFO:teuthology.orchestra.run.smithi050.stdout:(8/8): python3-scipy-1.0.0-20.module_el8.1.0+24 15 MB/s | 15 MB 00:00 2020-03-09T00:22:16.658 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:16.658 INFO:teuthology.orchestra.run.smithi050.stdout:Total 5.9 MB/s | 37 MB 00:06 2020-03-09T00:22:16.671 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:16.693 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:16.693 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:17.014 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:17.014 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:17.168 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-09T00:22:17.268 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:17.274 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-09T00:22:17.350 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:17.527 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-09T00:22:17.620 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:17.620 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 1/8 2020-03-09T00:22:17.621 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-09T00:22:17.621 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:17.621 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-09T00:22:17.621 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:17.621 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-09T00:22:17.621 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-09T00:22:17.687 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:17.711 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-09T00:22:17.831 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:17.905 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-09T00:22:17.905 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:17.906 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:17.906 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:17.906 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-09T00:22:17.906 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-09T00:22:17.906 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:17.906 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:17.906 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-09T00:22:17.907 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:17.907 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:17.907 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:17.907 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:17.911 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-09T00:22:17.985 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-09T00:22:18.087 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:18.091 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2020-03-09T00:22:18.125 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:18.293 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:18.293 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 1/13 2020-03-09T00:22:18.293 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-09T00:22:18.293 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-09T00:22:18.293 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-09T00:22:18.293 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-09T00:22:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-09T00:22:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-09T00:22:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-09T00:22:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-09T00:22:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-09T00:22:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-09T00:22:18.294 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-09T00:22:18.485 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:29 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:18.524 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-09T00:22:18.611 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-09T00:22:18.611 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:18.611 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:18.611 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:18.611 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-09T00:22:18.611 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-09T00:22:18.612 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-09T00:22:18.612 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-09T00:22:18.612 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-09T00:22:18.612 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-09T00:22:18.612 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-09T00:22:18.612 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-09T00:22:18.612 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-09T00:22:18.613 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-09T00:22:18.613 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-09T00:22:18.613 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-09T00:22:18.613 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:18.613 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:18.698 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:18.699 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-09T00:22:18.727 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-cephadm 2020-03-09T00:22:18.834 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:18.848 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:18.850 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:18.850 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2020-03-09T00:22:18.850 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:18.850 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:18.851 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 24 k 2020-03-09T00:22:18.851 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:22:18.851 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-2.el8 AppStream 27 k 2020-03-09T00:22:18.851 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 1.10-11.el8 AppStream 20 k 2020-03-09T00:22:18.851 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.3.7-6.el8 AppStream 126 k 2020-03-09T00:22:18.851 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 AppStream 110 k 2020-03-09T00:22:18.851 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 2.1.0-1.el8 BaseOS 155 k 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 BaseOS 43 k 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-09T00:22:18.852 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:18.853 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:18.853 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:18.853 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2020-03-09T00:22:18.853 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:18.854 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 1.9 M 2020-03-09T00:22:18.854 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 23 M 2020-03-09T00:22:18.854 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 1/8 2020-03-09T00:22:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-09T00:22:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-09T00:22:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-09T00:22:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-09T00:22:19.124 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:30 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:19.204 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-09T00:22:19.205 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:19.205 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:19.205 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:19.205 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-09T00:22:19.206 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-09T00:22:19.206 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:19.206 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:19.206 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-09T00:22:19.206 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:19.207 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:19.207 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:19.207 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:19.338 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 273 kB/s | 27 kB 00:00 2020-03-09T00:22:19.338 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 204 kB/s | 20 kB 00:00 2020-03-09T00:22:19.342 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-rook 2020-03-09T00:22:19.351 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): ceph-mgr-rook-15.1.0-1904.g9eb9490.el8. 220 kB/s | 24 kB 00:00 2020-03-09T00:22:19.406 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:19.508 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 986 kB/s | 155 kB 00:00 2020-03-09T00:22:19.513 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 719 kB/s | 126 kB 00:00 2020-03-09T00:22:19.530 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 573 kB/s | 110 kB 00:00 2020-03-09T00:22:19.537 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:19.538 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:19.538 INFO:teuthology.orchestra.run.smithi139.stdout: Package ArchVersion Repository Size 2020-03-09T00:22:19.539 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:19.539 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:19.539 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 36 k 2020-03-09T00:22:19.539 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:22:19.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-09T00:22:19.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-09T00:22:19.539 INFO:teuthology.orchestra.run.smithi139.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-09T00:22:19.540 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:19.540 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:19.540 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:19.540 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2020-03-09T00:22:19.540 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:19.541 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2020-03-09T00:22:19.541 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 461 k 2020-03-09T00:22:19.541 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 1.4 M 2020-03-09T00:22:19.541 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:19.559 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:19.561 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 689 kB/s | 33 kB 00:00 2020-03-09T00:22:19.587 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 289 kB/s | 16 kB 00:00 2020-03-09T00:22:19.593 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 920 kB/s | 48 kB 00:00 2020-03-09T00:22:19.608 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.1 MB/s | 356 kB 00:00 2020-03-09T00:22:19.615 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.1 MB/s | 81 kB 00:00 2020-03-09T00:22:19.625 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-rsa-3.4.2-12.el8.noarch.rpm 1.9 MB/s | 73 kB 00:00 2020-03-09T00:22:19.690 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 374 kB/s | 21 kB 00:00 2020-03-09T00:22:19.690 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1904.g9eb9490.el 325 kB/s | 36 kB 00:00 2020-03-09T00:22:19.691 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:19.691 INFO:teuthology.orchestra.run.smithi139.stdout:Total 4.0 MB/s | 461 kB 00:00 2020-03-09T00:22:19.691 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:19.691 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:19.691 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:19.691 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:19.692 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:19.692 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 3.1 MB/s | 61 kB 00:00 2020-03-09T00:22:19.692 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 12 MB/s | 1.1 MB 00:00 2020-03-09T00:22:19.692 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:19.692 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.3 MB/s | 1.9 MB 00:00 2020-03-09T00:22:19.692 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:19.692 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:19.693 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:19.738 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:30 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:19.754 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:19.755 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:19.866 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:20.114 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:20.116 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:20.116 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2020-03-09T00:22:20.116 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:20.116 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:20.117 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 24 k 2020-03-09T00:22:20.117 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:22:20.117 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-2.el8 AppStream 27 k 2020-03-09T00:22:20.117 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 1.10-11.el8 AppStream 20 k 2020-03-09T00:22:20.117 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.3.7-6.el8 AppStream 126 k 2020-03-09T00:22:20.117 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 AppStream 110 k 2020-03-09T00:22:20.117 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 2.1.0-1.el8 BaseOS 155 k 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 BaseOS 43 k 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-09T00:22:20.118 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:20.119 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:20.119 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:20.119 INFO:teuthology.orchestra.run.smithi148.stdout:Install 13 Packages 2020-03-09T00:22:20.119 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:20.120 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 1.9 M 2020-03-09T00:22:20.120 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 23 M 2020-03-09T00:22:20.120 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:20.291 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-09T00:22:20.302 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:20.535 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-09T00:22:20.545 INFO:teuthology.orchestra.run.smithi148.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 300 kB/s | 27 kB 00:00 2020-03-09T00:22:20.554 INFO:teuthology.orchestra.run.smithi148.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 205 kB/s | 20 kB 00:00 2020-03-09T00:22:20.608 INFO:teuthology.orchestra.run.smithi148.stdout:(3/13): ceph-mgr-rook-15.1.0-1904.g9eb9490.el8. 160 kB/s | 24 kB 00:00 2020-03-09T00:22:20.730 INFO:teuthology.orchestra.run.smithi148.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 628 kB/s | 110 kB 00:00 2020-03-09T00:22:20.735 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:20.738 INFO:teuthology.orchestra.run.smithi148.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 655 kB/s | 126 kB 00:00 2020-03-09T00:22:20.754 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-09T00:22:20.766 INFO:teuthology.orchestra.run.smithi148.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 986 kB/s | 155 kB 00:00 2020-03-09T00:22:20.786 INFO:teuthology.orchestra.run.smithi148.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 692 kB/s | 33 kB 00:00 2020-03-09T00:22:20.817 INFO:teuthology.orchestra.run.smithi148.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 2.6 MB/s | 81 kB 00:00 2020-03-09T00:22:20.822 INFO:teuthology.orchestra.run.smithi148.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 289 kB/s | 16 kB 00:00 2020-03-09T00:22:20.851 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-09T00:22:20.854 INFO:teuthology.orchestra.run.smithi148.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 342 kB/s | 43 kB 00:00 2020-03-09T00:22:20.860 INFO:teuthology.orchestra.run.smithi148.stdout:(11/13): python3-rsa-3.4.2-12.el8.noarch.rpm 1.9 MB/s | 73 kB 00:00 2020-03-09T00:22:20.911 INFO:teuthology.orchestra.run.smithi148.stdout:(12/13): python3-kubernetes-10.0.1-2.el8.noarch 12 MB/s | 1.1 MB 00:00 2020-03-09T00:22:20.930 INFO:teuthology.orchestra.run.smithi148.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 814 kB/s | 61 kB 00:00 2020-03-09T00:22:20.930 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:20.930 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.3 MB/s | 1.9 MB 00:00 2020-03-09T00:22:20.937 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:20.952 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:20.952 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:21.046 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:21.046 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:21.075 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:21.125 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:21.351 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:21.520 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:21.590 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:21.726 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:21.727 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 1/4 2020-03-09T00:22:21.727 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:21.727 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:21.808 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-09T00:22:21.897 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-09T00:22:21.969 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-09T00:22:21.969 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:21.969 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:21.969 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:21.969 INFO:teuthology.orchestra.run.smithi139.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-09T00:22:21.970 INFO:teuthology.orchestra.run.smithi139.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-09T00:22:21.970 INFO:teuthology.orchestra.run.smithi139.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-09T00:22:21.970 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:21.970 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:22.082 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-09T00:22:22.084 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-fuse 2020-03-09T00:22:22.107 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-09T00:22:22.323 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-09T00:22:22.482 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:33 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:22.526 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-09T00:22:22.573 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-09T00:22:22.721 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-09T00:22:22.850 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-09T00:22:22.884 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:22.885 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2020-03-09T00:22:22.885 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:22.885 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:22.885 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 572 k 2020-03-09T00:22:22.885 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2020-03-09T00:22:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.7-12.el8 BaseOS 82 k 2020-03-09T00:22:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-common x86_64 3.2.1-12.el8 BaseOS 21 k 2020-03-09T00:22:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:22.886 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:22.886 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:22.886 INFO:teuthology.orchestra.run.smithi139.stdout:Install 3 Packages 2020-03-09T00:22:22.886 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:22.887 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 676 k 2020-03-09T00:22:22.888 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 2.0 M 2020-03-09T00:22:22.888 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:22.889 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-09T00:22:23.063 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-09T00:22:23.114 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-09T00:22:23.160 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-09T00:22:23.192 INFO:teuthology.orchestra.run.smithi139.stdout:(1/3): fuse-common-3.2.1-12.el8.x86_64.rpm 200 kB/s | 21 kB 00:00 2020-03-09T00:22:23.224 INFO:teuthology.orchestra.run.smithi139.stdout:(2/3): fuse-2.9.7-12.el8.x86_64.rpm 603 kB/s | 82 kB 00:00 2020-03-09T00:22:23.279 INFO:teuthology.orchestra.run.smithi139.stdout:(3/3): ceph-fuse-15.1.0-1904.g9eb9490.el8.x86_6 2.9 MB/s | 572 kB 00:00 2020-03-09T00:22:23.279 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:23.280 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.7 MB/s | 676 kB 00:00 2020-03-09T00:22:23.283 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:23.304 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:23.304 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:23.339 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-09T00:22:23.367 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:23.367 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:23.528 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-09T00:22:23.573 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:23.573 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-09T00:22:23.573 INFO:teuthology.orchestra.run.smithi139.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-09T00:22:23.574 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:23.993 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:24.125 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-09T00:22:24.160 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:24.363 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-09T00:22:24.434 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:24.434 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/3 2020-03-09T00:22:24.435 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:24.547 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-09T00:22:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-09T00:22:24.643 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-09T00:22:24.747 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-09T00:22:24.787 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-09T00:22:24.798 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-09T00:22:24.798 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:24.798 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:24.799 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-09T00:22:24.799 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-09T00:22:24.799 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:24.799 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:24.931 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:24.946 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-09T00:22:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:25.088 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-09T00:22:25.103 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install librados-devel 2020-03-09T00:22:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 1/13 2020-03-09T00:22:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-09T00:22:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-09T00:22:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-09T00:22:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-09T00:22:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-09T00:22:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-09T00:22:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-09T00:22:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-09T00:22:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-09T00:22:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-09T00:22:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-09T00:22:25.239 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:25.251 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 8/8 2020-03-09T00:22:25.251 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490 1/8 2020-03-09T00:22:25.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-09T00:22:25.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-09T00:22:25.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-09T00:22:25.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-09T00:22:25.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-09T00:22:25.252 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-09T00:22:25.446 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:25.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 1/13 2020-03-09T00:22:25.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-09T00:22:25.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-09T00:22:25.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-09T00:22:25.446 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-09T00:22:25.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-09T00:22:25.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-09T00:22:25.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-09T00:22:25.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-09T00:22:25.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-09T00:22:25.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-09T00:22:25.447 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-09T00:22:25.508 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:36 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:25.512 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-09T00:22:25.512 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:25.512 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:25.512 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:25.512 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-09T00:22:25.512 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-09T00:22:25.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-09T00:22:25.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-09T00:22:25.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-09T00:22:25.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-09T00:22:25.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-09T00:22:25.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-09T00:22:25.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-09T00:22:25.514 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-09T00:22:25.514 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-09T00:22:25.514 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-09T00:22:25.514 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:25.514 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:25.656 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2020-03-09T00:22:25.676 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-09T00:22:25.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:25.676 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:25.676 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:25.676 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-09T00:22:25.676 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-09T00:22:25.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:25.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-09T00:22:25.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-09T00:22:25.677 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:25.677 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-09T00:22:25.677 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:25.677 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:25.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-09T00:22:25.776 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:25.776 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:25.776 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:25.776 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-09T00:22:25.776 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-09T00:22:25.776 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-09T00:22:25.776 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-09T00:22:25.777 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-09T00:22:25.777 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-09T00:22:25.777 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-09T00:22:25.777 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-09T00:22:25.777 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-09T00:22:25.777 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-09T00:22:25.777 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-09T00:22:25.778 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-09T00:22:25.778 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:25.778 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:25.807 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2020-03-09T00:22:25.905 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-cephadm 2020-03-09T00:22:25.911 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:25.912 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:25.913 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2020-03-09T00:22:25.913 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:25.913 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:25.913 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:25.914 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:25.914 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:25.914 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:25.914 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2020-03-09T00:22:25.914 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:25.915 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 141 k 2020-03-09T00:22:25.915 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 484 k 2020-03-09T00:22:25.915 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:26.052 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:37 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:26.103 INFO:teuthology.orchestra.run.smithi139.stdout:librados-devel-15.1.0-1904.g9eb9490.el8.x86_64. 739 kB/s | 141 kB 00:00 2020-03-09T00:22:26.104 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:26.105 INFO:teuthology.orchestra.run.smithi139.stdout:Total 733 kB/s | 141 kB 00:00 2020-03-09T00:22:26.106 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:26.114 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:26.114 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:26.163 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:26.163 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:26.233 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:37 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:26.305 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:37 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:26.411 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:26.412 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:26.412 INFO:teuthology.orchestra.run.smithi104.stdout: Package ArchVersion Repository Size 2020-03-09T00:22:26.412 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:26.412 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:26.412 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 36 k 2020-03-09T00:22:26.413 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:22:26.413 INFO:teuthology.orchestra.run.smithi104.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-09T00:22:26.413 INFO:teuthology.orchestra.run.smithi104.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-09T00:22:26.413 INFO:teuthology.orchestra.run.smithi104.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-09T00:22:26.413 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:26.413 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:26.413 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:26.414 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2020-03-09T00:22:26.414 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:26.414 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 461 k 2020-03-09T00:22:26.414 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 1.4 M 2020-03-09T00:22:26.415 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:26.469 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 869 kB/s | 48 kB 00:00 2020-03-09T00:22:26.486 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 4.8 MB/s | 356 kB 00:00 2020-03-09T00:22:26.517 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 443 kB/s | 21 kB 00:00 2020-03-09T00:22:26.529 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1904.g9eb9490.el 308 kB/s | 36 kB 00:00 2020-03-09T00:22:26.531 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:26.532 INFO:teuthology.orchestra.run.smithi104.stdout:Total 3.8 MB/s | 461 kB 00:00 2020-03-09T00:22:26.535 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:26.542 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:26.542 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:26.573 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:26.573 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:26.589 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:26.619 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:26.621 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:26.621 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2020-03-09T00:22:26.621 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:26.621 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:15.1.0-1904.g9eb9490.el8 ceph-noarch 24 k 2020-03-09T00:22:26.622 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:22:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-2.el8 AppStream 27 k 2020-03-09T00:22:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 1.10-11.el8 AppStream 20 k 2020-03-09T00:22:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.3.7-6.el8 AppStream 126 k 2020-03-09T00:22:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 AppStream 110 k 2020-03-09T00:22:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 2.1.0-1.el8 BaseOS 155 k 2020-03-09T00:22:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 BaseOS 43 k 2020-03-09T00:22:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-09T00:22:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-09T00:22:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-09T00:22:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-09T00:22:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-09T00:22:26.623 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-09T00:22:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:26.624 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:26.624 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:26.624 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2020-03-09T00:22:26.624 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:26.625 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.9 M 2020-03-09T00:22:26.625 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 23 M 2020-03-09T00:22:26.625 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:26.687 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:26.688 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:26.689 INFO:teuthology.orchestra.run.smithi148.stdout: Package ArchVersion Repository Size 2020-03-09T00:22:26.689 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:26.689 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:26.689 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 36 k 2020-03-09T00:22:26.689 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:22:26.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-09T00:22:26.689 INFO:teuthology.orchestra.run.smithi148.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-09T00:22:26.690 INFO:teuthology.orchestra.run.smithi148.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-09T00:22:26.690 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:26.690 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:26.690 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:26.690 INFO:teuthology.orchestra.run.smithi148.stdout:Install 4 Packages 2020-03-09T00:22:26.690 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:26.691 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 461 k 2020-03-09T00:22:26.691 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 1.4 M 2020-03-09T00:22:26.691 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:26.744 INFO:teuthology.orchestra.run.smithi148.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 891 kB/s | 48 kB 00:00 2020-03-09T00:22:26.752 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:26.758 INFO:teuthology.orchestra.run.smithi148.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.1 MB/s | 356 kB 00:00 2020-03-09T00:22:26.791 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:26.800 INFO:teuthology.orchestra.run.smithi148.stdout:(3/4): ceph-mgr-cephadm-15.1.0-1904.g9eb9490.el 323 kB/s | 36 kB 00:00 2020-03-09T00:22:26.801 INFO:teuthology.orchestra.run.smithi148.stdout:(4/4): python3-apipkg-1.5-6.el8.noarch.rpm 375 kB/s | 21 kB 00:00 2020-03-09T00:22:26.803 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:26.803 INFO:teuthology.orchestra.run.smithi148.stdout:Total 4.0 MB/s | 461 kB 00:00 2020-03-09T00:22:26.806 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:26.814 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:26.814 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:26.839 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:26.840 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:26.962 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 273 kB/s | 27 kB 00:00 2020-03-09T00:22:26.963 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 204 kB/s | 20 kB 00:00 2020-03-09T00:22:26.974 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): ceph-mgr-rook-15.1.0-1904.g9eb9490.el8. 222 kB/s | 24 kB 00:00 2020-03-09T00:22:26.989 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:26.990 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-09T00:22:26.994 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:27.132 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 984 kB/s | 155 kB 00:00 2020-03-09T00:22:27.155 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 573 kB/s | 110 kB 00:00 2020-03-09T00:22:27.156 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 653 kB/s | 126 kB 00:00 2020-03-09T00:22:27.164 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2020-03-09T00:22:27.203 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 351 kB/s | 16 kB 00:00 2020-03-09T00:22:27.213 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 571 kB/s | 33 kB 00:00 2020-03-09T00:22:27.217 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:27.217 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:27.217 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:27.217 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:27.217 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:27.218 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:27.225 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.3 MB/s | 81 kB 00:00 2020-03-09T00:22:27.231 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-09T00:22:27.241 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.8 MB/s | 61 kB 00:00 2020-03-09T00:22:27.251 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-rsa-3.4.2-12.el8.noarch.rpm 1.9 MB/s | 73 kB 00:00 2020-03-09T00:22:27.312 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 10 MB/s | 1.1 MB 00:00 2020-03-09T00:22:27.312 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:27.313 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.7 MB/s | 1.9 MB 00:00 2020-03-09T00:22:27.321 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:27.337 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:27.337 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:27.409 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:27.409 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:27.458 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:27.494 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs2 2020-03-09T00:22:27.629 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:27.870 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:27.896 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:38 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:28.039 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:28.054 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:28.172 INFO:teuthology.orchestra.run.smithi139.stdout:Package libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:28.240 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:28.240 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 1/4 2020-03-09T00:22:28.240 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:28.240 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:28.279 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:28.279 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:22:28.280 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:28.438 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:28.480 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs-devel 2020-03-09T00:22:28.511 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-09T00:22:28.511 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:28.511 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:28.512 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:28.512 INFO:teuthology.orchestra.run.smithi104.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-09T00:22:28.512 INFO:teuthology.orchestra.run.smithi104.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-09T00:22:28.512 INFO:teuthology.orchestra.run.smithi104.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-09T00:22:28.512 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:28.512 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:28.642 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:28.642 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 1/4 2020-03-09T00:22:28.642 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:28.643 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:28.655 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2020-03-09T00:22:28.877 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:39 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:28.886 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-09T00:22:28.886 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:28.886 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:28.886 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:28.886 INFO:teuthology.orchestra.run.smithi148.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-09T00:22:28.886 INFO:teuthology.orchestra.run.smithi148.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-09T00:22:28.887 INFO:teuthology.orchestra.run.smithi148.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-09T00:22:28.887 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:28.887 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:29.015 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-fuse 2020-03-09T00:22:29.051 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:40 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:29.057 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:29.260 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:29.260 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:29.261 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2020-03-09T00:22:29.261 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:29.261 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:29.261 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 k 2020-03-09T00:22:29.261 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:29.261 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:29.261 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:29.262 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2020-03-09T00:22:29.262 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:29.262 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 21 k 2020-03-09T00:22:29.262 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 76 k 2020-03-09T00:22:29.262 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:29.331 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-09T00:22:29.373 INFO:teuthology.orchestra.run.smithi139.stdout:libcephfs-devel-15.1.0-1904.g9eb9490.el8.x86_64 191 kB/s | 21 kB 00:00 2020-03-09T00:22:29.373 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:29.374 INFO:teuthology.orchestra.run.smithi139.stdout:Total 190 kB/s | 21 kB 00:00 2020-03-09T00:22:29.375 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:29.379 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:29.379 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:29.382 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:29.382 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:29.404 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:29.405 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:29.405 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2020-03-09T00:22:29.406 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:29.406 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:29.406 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 572 k 2020-03-09T00:22:29.406 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2020-03-09T00:22:29.407 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.7-12.el8 BaseOS 82 k 2020-03-09T00:22:29.407 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-common x86_64 3.2.1-12.el8 BaseOS 21 k 2020-03-09T00:22:29.407 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:29.407 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:29.408 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:29.408 INFO:teuthology.orchestra.run.smithi104.stdout:Install 3 Packages 2020-03-09T00:22:29.408 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:29.408 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 676 k 2020-03-09T00:22:29.409 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 2.0 M 2020-03-09T00:22:29.409 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:29.415 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:40 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:29.427 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:29.567 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-09T00:22:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:29.692 INFO:teuthology.orchestra.run.smithi104.stdout:(1/3): fuse-common-3.2.1-12.el8.x86_64.rpm 226 kB/s | 21 kB 00:00 2020-03-09T00:22:29.723 INFO:teuthology.orchestra.run.smithi104.stdout:(2/3): fuse-2.9.7-12.el8.x86_64.rpm 661 kB/s | 82 kB 00:00 2020-03-09T00:22:29.776 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:29.777 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:29.778 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:29.779 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2020-03-09T00:22:29.779 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:29.779 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:29.779 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 572 k 2020-03-09T00:22:29.780 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2020-03-09T00:22:29.780 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.7-12.el8 BaseOS 82 k 2020-03-09T00:22:29.780 INFO:teuthology.orchestra.run.smithi148.stdout: fuse-common x86_64 3.2.1-12.el8 BaseOS 21 k 2020-03-09T00:22:29.780 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:29.780 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:29.781 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:29.781 INFO:teuthology.orchestra.run.smithi148.stdout:Install 3 Packages 2020-03-09T00:22:29.781 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:29.782 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 676 k 2020-03-09T00:22:29.782 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 2.0 M 2020-03-09T00:22:29.783 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:29.789 INFO:teuthology.orchestra.run.smithi104.stdout:(3/3): ceph-fuse-15.1.0-1904.g9eb9490.el8.x86_6 2.9 MB/s | 572 kB 00:00 2020-03-09T00:22:29.791 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:29.791 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.7 MB/s | 676 kB 00:00 2020-03-09T00:22:29.794 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:29.804 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-09T00:22:29.817 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:29.817 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:29.879 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:29.880 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:30.008 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:30.009 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:30.009 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:30.009 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:30.032 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-09T00:22:30.076 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:30.076 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-09T00:22:30.076 INFO:teuthology.orchestra.run.smithi104.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-09T00:22:30.076 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:30.123 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install librados2 2020-03-09T00:22:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-09T00:22:30.369 INFO:teuthology.orchestra.run.smithi148.stdout:(1/3): fuse-common-3.2.1-12.el8.x86_64.rpm 225 kB/s | 21 kB 00:00 2020-03-09T00:22:30.402 INFO:teuthology.orchestra.run.smithi148.stdout:(2/3): fuse-2.9.7-12.el8.x86_64.rpm 646 kB/s | 82 kB 00:00 2020-03-09T00:22:30.467 INFO:teuthology.orchestra.run.smithi148.stdout:(3/3): ceph-fuse-15.1.0-1904.g9eb9490.el8.x86_6 2.9 MB/s | 572 kB 00:00 2020-03-09T00:22:30.468 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:30.468 INFO:teuthology.orchestra.run.smithi148.stdout:Total 981 kB/s | 676 kB 00:00 2020-03-09T00:22:30.472 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:30.493 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:30.494 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:30.505 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:30.520 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:41 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:30.526 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-09T00:22:30.558 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:30.559 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:30.687 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:30.721 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-09T00:22:30.721 INFO:teuthology.orchestra.run.smithi148.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-09T00:22:30.721 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:30.771 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-09T00:22:30.876 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:30.877 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:22:30.878 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:30.976 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:30.976 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/3 2020-03-09T00:22:30.976 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:31.014 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install librbd1 2020-03-09T00:22:31.065 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-09T00:22:31.137 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:31.277 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-09T00:22:31.277 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:31.277 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:31.278 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-09T00:22:31.278 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-09T00:22:31.278 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:31.278 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:31.287 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:31.412 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:42 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:31.566 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2020-03-09T00:22:31.586 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:31.586 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/3 2020-03-09T00:22:31.586 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:31.671 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:31.775 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:31.777 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:22:31.777 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:31.920 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-09T00:22:31.921 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:31.921 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:31.921 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-09T00:22:31.921 INFO:teuthology.orchestra.run.smithi148.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-09T00:22:31.921 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:31.921 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:31.967 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:42 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:32.183 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install librados-devel 2020-03-09T00:22:32.211 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-09T00:22:32.382 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rados 2020-03-09T00:22:32.392 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:32.392 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:32.392 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2020-03-09T00:22:32.393 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:32.394 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 141 k 2020-03-09T00:22:32.394 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 484 k 2020-03-09T00:22:32.394 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:32.481 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-09T00:22:32.545 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-15.1.0-1904.g9eb9490.el8.x86_64. 929 kB/s | 141 kB 00:00 2020-03-09T00:22:32.546 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:32.546 INFO:teuthology.orchestra.run.smithi104.stdout:Total 922 kB/s | 141 kB 00:00 2020-03-09T00:22:32.548 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:32.554 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:32.554 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:32.593 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:43 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:32.603 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:32.604 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:32.631 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-09T00:22:32.783 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:43 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:32.823 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-09T00:22:33.002 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:33.003 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:33.003 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2020-03-09T00:22:33.003 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:33.003 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:33.003 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:33.003 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:33.003 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:33.004 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:33.004 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2020-03-09T00:22:33.004 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:33.004 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 141 k 2020-03-09T00:22:33.005 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 484 k 2020-03-09T00:22:33.005 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:33.025 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:33.039 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:33.054 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:33.157 INFO:teuthology.orchestra.run.smithi148.stdout:librados-devel-15.1.0-1904.g9eb9490.el8.x86_64. 926 kB/s | 141 kB 00:00 2020-03-09T00:22:33.157 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:33.157 INFO:teuthology.orchestra.run.smithi148.stdout:Total 919 kB/s | 141 kB 00:00 2020-03-09T00:22:33.159 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:33.165 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:33.166 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:33.166 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:33.166 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:22:33.166 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:33.215 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:33.215 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:33.230 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 13/13 2020-03-09T00:22:33.231 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 1/13 2020-03-09T00:22:33.231 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-09T00:22:33.231 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-09T00:22:33.231 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-09T00:22:33.231 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-09T00:22:33.231 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-09T00:22:33.231 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-09T00:22:33.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-09T00:22:33.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-09T00:22:33.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-09T00:22:33.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-09T00:22:33.232 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-09T00:22:33.277 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:33.325 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:33.338 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rgw 2020-03-09T00:22:33.466 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:33.510 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-09T00:22:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:33.569 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-09T00:22:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-09T00:22:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-09T00:22:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-09T00:22:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-09T00:22:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-09T00:22:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-09T00:22:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-09T00:22:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-09T00:22:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-09T00:22:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-09T00:22:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-09T00:22:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:33.571 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:33.724 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:33.730 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2020-03-09T00:22:33.735 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:44 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:33.736 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:33.736 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:33.736 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:34.004 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:34.006 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2020-03-09T00:22:34.109 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:34.110 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:22:34.110 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:34.160 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:45 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:34.244 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-cephfs 2020-03-09T00:22:34.414 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:45 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:34.416 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:34.416 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:34.416 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:34.416 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:34.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:34.417 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:34.559 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:34.560 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: Package ArchVersion Repository Size 2020-03-09T00:22:34.560 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:34.560 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch2:15.1.0-1904.g9eb9490.el8 ceph-noarch 36 k 2020-03-09T00:22:34.560 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:22:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-09T00:22:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-09T00:22:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-09T00:22:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:34.561 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:34.562 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:34.562 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2020-03-09T00:22:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:34.563 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 461 k 2020-03-09T00:22:34.563 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.4 M 2020-03-09T00:22:34.563 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:34.618 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 860 kB/s | 48 kB 00:00 2020-03-09T00:22:34.630 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.1 MB/s | 356 kB 00:00 2020-03-09T00:22:34.640 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:45 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:34.672 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): ceph-mgr-cephadm-15.1.0-1904.g9eb9490.el 322 kB/s | 36 kB 00:00 2020-03-09T00:22:34.675 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): python3-apipkg-1.5-6.el8.noarch.rpm 369 kB/s | 21 kB 00:00 2020-03-09T00:22:34.677 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:34.677 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.9 MB/s | 461 kB 00:00 2020-03-09T00:22:34.681 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:34.689 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:34.689 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:34.703 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:34.740 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:34.740 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:34.814 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:34.814 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:34.815 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:34.893 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:34.904 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:34.922 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2020-03-09T00:22:35.009 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:35.010 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs2 2020-03-09T00:22:35.011 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:22:35.011 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:35.128 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rbd 2020-03-09T00:22:35.164 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-09T00:22:35.320 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:46 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:35.390 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:35.422 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:46 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:35.526 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:46 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:35.604 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:35.680 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:35.681 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:35.681 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2020-03-09T00:22:35.681 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:35.682 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:35.682 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 k 2020-03-09T00:22:35.682 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:35.682 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:35.682 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:35.682 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2020-03-09T00:22:35.683 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:35.683 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 21 k 2020-03-09T00:22:35.683 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 76 k 2020-03-09T00:22:35.683 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:35.705 INFO:teuthology.orchestra.run.smithi148.stdout:Package libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:35.787 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:35.792 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-15.1.0-1904.g9eb9490.el8.x86_64 193 kB/s | 21 kB 00:00 2020-03-09T00:22:35.793 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:35.793 INFO:teuthology.orchestra.run.smithi104.stdout:Total 192 kB/s | 21 kB 00:00 2020-03-09T00:22:35.795 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:35.799 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:35.799 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:35.801 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:35.801 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:35.812 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:35.813 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:35.813 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:35.846 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:35.894 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:35.895 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2020-03-09T00:22:35.895 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:35.931 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs-devel 2020-03-09T00:22:36.002 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 4/4 2020-03-09T00:22:36.002 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 1/4 2020-03-09T00:22:36.003 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-09T00:22:36.003 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-09T00:22:36.011 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:36.030 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-fuse 2020-03-09T00:22:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:36.260 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-09T00:22:36.261 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:36.261 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:36.261 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:15.1.0-1904.g9eb9490.el8.noarch 2020-03-09T00:22:36.261 INFO:teuthology.orchestra.run.smithi050.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-09T00:22:36.261 INFO:teuthology.orchestra.run.smithi050.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-09T00:22:36.262 INFO:teuthology.orchestra.run.smithi050.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-09T00:22:36.262 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:36.262 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:36.330 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:47 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:36.381 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2020-03-09T00:22:36.426 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:36.426 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:36.426 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:36.426 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:36.426 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:36.427 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:36.430 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:47 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:36.564 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2020-03-09T00:22:36.697 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:36.697 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:36.697 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2020-03-09T00:22:36.697 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:36.698 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:36.698 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 k 2020-03-09T00:22:36.698 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:36.698 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:36.698 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:36.698 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2020-03-09T00:22:36.698 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:36.699 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 21 k 2020-03-09T00:22:36.699 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 76 k 2020-03-09T00:22:36.699 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:36.809 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:47 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:36.810 INFO:teuthology.orchestra.run.smithi148.stdout:libcephfs-devel-15.1.0-1904.g9eb9490.el8.x86_64 192 kB/s | 21 kB 00:00 2020-03-09T00:22:36.810 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:36.810 INFO:teuthology.orchestra.run.smithi148.stdout:Total 190 kB/s | 21 kB 00:00 2020-03-09T00:22:36.811 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:36.811 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:36.811 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:36.811 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:36.811 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:36.812 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 72 k 2020-03-09T00:22:36.812 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:36.812 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:36.812 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:36.812 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2020-03-09T00:22:36.812 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:36.812 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 72 k 2020-03-09T00:22:36.813 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 215 k 2020-03-09T00:22:36.813 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:36.813 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:36.816 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:36.816 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:36.818 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:36.818 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:36.876 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:36.942 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-fuse-15.1.0-1904.g9eb9490.el8.x86_64.rpm 551 kB/s | 72 kB 00:00 2020-03-09T00:22:36.943 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:36.943 INFO:teuthology.orchestra.run.smithi139.stdout:Total 547 kB/s | 72 kB 00:00 2020-03-09T00:22:36.945 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:36.955 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:36.956 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:36.962 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:47 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:37.002 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:37.003 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:37.004 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:37.106 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:37.182 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:37.192 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:37.193 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:37.193 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2020-03-09T00:22:37.193 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:37.194 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:37.194 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 572 k 2020-03-09T00:22:37.194 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2020-03-09T00:22:37.194 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.7-12.el8 BaseOS 82 k 2020-03-09T00:22:37.194 INFO:teuthology.orchestra.run.smithi050.stdout: fuse-common x86_64 3.2.1-12.el8 BaseOS 21 k 2020-03-09T00:22:37.194 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:37.194 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:37.195 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:37.195 INFO:teuthology.orchestra.run.smithi050.stdout:Install 3 Packages 2020-03-09T00:22:37.195 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:37.195 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 676 k 2020-03-09T00:22:37.195 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.0 M 2020-03-09T00:22:37.196 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:37.218 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:37.290 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:37.323 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:37.324 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:37.324 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:37.378 INFO:teuthology.orchestra.run.smithi050.stdout:(1/3): fuse-common-3.2.1-12.el8.x86_64.rpm 225 kB/s | 21 kB 00:00 2020-03-09T00:22:37.409 INFO:teuthology.orchestra.run.smithi050.stdout:(2/3): fuse-2.9.7-12.el8.x86_64.rpm 660 kB/s | 82 kB 00:00 2020-03-09T00:22:37.426 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:37.426 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:37.426 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:37.426 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:37.426 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:37.426 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:37.444 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2020-03-09T00:22:37.479 INFO:teuthology.orchestra.run.smithi050.stdout:(3/3): ceph-fuse-15.1.0-1904.g9eb9490.el8.x86_6 2.9 MB/s | 572 kB 00:00 2020-03-09T00:22:37.480 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:37.480 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.3 MB/s | 676 kB 00:00 2020-03-09T00:22:37.485 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:37.507 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:37.507 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:37.555 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install librados2 2020-03-09T00:22:37.576 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:37.577 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:37.786 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:37.786 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-09T00:22:37.786 INFO:teuthology.orchestra.run.smithi050.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-09T00:22:37.786 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:37.840 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:48 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:37.954 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:48 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:37.982 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:38.101 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:38.206 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:38.207 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:38.207 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:38.221 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:38.325 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:38.326 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:38.327 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:38.372 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:38.373 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:38.373 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:38.373 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:38.373 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:38.373 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:38.405 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:38.488 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install librbd1 2020-03-09T00:22:38.562 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2020-03-09T00:22:38.641 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-mirror 2020-03-09T00:22:38.728 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 3/3 2020-03-09T00:22:38.729 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/3 2020-03-09T00:22:38.729 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-09T00:22:38.889 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:49 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:38.962 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:49 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:39.040 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:50 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-09T00:22:39.048 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:39.048 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-09T00:22:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-09T00:22:39.049 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:39.049 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:39.311 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:39.311 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:39.311 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:39.312 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:39.312 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:39.341 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:39.342 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:39.342 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:39.389 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rados 2020-03-09T00:22:39.413 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2020-03-09T00:22:39.435 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:39.435 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:39.435 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2020-03-09T00:22:39.435 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:39.435 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:39.435 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 2.7 M 2020-03-09T00:22:39.436 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:39.436 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:39.436 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:39.436 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2020-03-09T00:22:39.436 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:39.436 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 2.7 M 2020-03-09T00:22:39.436 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 12 M 2020-03-09T00:22:39.437 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:39.462 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2020-03-09T00:22:39.793 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:50 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:39.831 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-mirror-15.1.0-1904.g9eb9490.el8.x86_64.rpm 6.7 MB/s | 2.7 MB 00:00 2020-03-09T00:22:39.832 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:39.832 INFO:teuthology.orchestra.run.smithi139.stdout:Total 6.7 MB/s | 2.7 MB 00:00 2020-03-09T00:22:39.834 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:39.850 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:39.851 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:39.857 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:50 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:39.858 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:50 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:39.908 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:39.908 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:40.117 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:40.188 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:40.223 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:40.224 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:40.224 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:40.269 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:40.300 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:40.301 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:40.301 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:40.317 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:40.318 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:40.318 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2020-03-09T00:22:40.318 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:40.319 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:40.319 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:40.319 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:40.319 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:40.319 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:40.319 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2020-03-09T00:22:40.319 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:40.320 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 141 k 2020-03-09T00:22:40.320 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 484 k 2020-03-09T00:22:40.320 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:40.328 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2020-03-09T00:22:40.421 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rgw 2020-03-09T00:22:40.470 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-15.1.0-1904.g9eb9490.el8.x86_64. 933 kB/s | 141 kB 00:00 2020-03-09T00:22:40.471 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:40.471 INFO:teuthology.orchestra.run.smithi050.stdout:Total 926 kB/s | 141 kB 00:00 2020-03-09T00:22:40.473 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:40.480 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:40.480 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:40.525 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:40.534 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:40.534 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:40.648 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:40.725 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:51 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:40.827 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:40.828 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:51 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:40.828 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:40.992 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:41.090 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:41.091 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:41.091 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:41.091 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:41.091 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:41.097 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:41.098 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:41.098 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:41.101 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:41.196 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:41.197 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:41.198 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:41.212 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2020-03-09T00:22:41.339 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-cephfs 2020-03-09T00:22:41.366 INFO:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-nbd 2020-03-09T00:22:41.410 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:41.411 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:41.411 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:41.411 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:41.411 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:41.411 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:41.607 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:52 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:41.700 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2020-03-09T00:22:41.740 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:52 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:41.771 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:52 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:41.875 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:41.981 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:41.982 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2020-03-09T00:22:41.982 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:42.012 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:42.076 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2020-03-09T00:22:42.118 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:42.119 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:42.119 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:42.139 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:53 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:42.146 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2020-03-09T00:22:42.147 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:42.147 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:42.147 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:42.148 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2020-03-09T00:22:42.148 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:42.148 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:42.148 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2020-03-09T00:22:42.148 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2020-03-09T00:22:42.148 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2020-03-09T00:22:42.148 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:42.149 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 141 k 2020-03-09T00:22:42.149 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 504 k 2020-03-09T00:22:42.149 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2020-03-09T00:22:42.238 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rbd 2020-03-09T00:22:42.298 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-nbd-15.1.0-1904.g9eb9490.el8.x86_64.rpm 934 kB/s | 141 kB 00:00 2020-03-09T00:22:42.299 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:42.299 INFO:teuthology.orchestra.run.smithi139.stdout:Total 927 kB/s | 141 kB 00:00 2020-03-09T00:22:42.301 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2020-03-09T00:22:42.312 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2020-03-09T00:22:42.313 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2020-03-09T00:22:42.361 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2020-03-09T00:22:42.362 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2020-03-09T00:22:42.446 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:42.467 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2020-03-09T00:22:42.472 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:53 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:42.557 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:42.558 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:42.558 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:42.640 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:53 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:42.672 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:42.695 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2020-03-09T00:22:42.843 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:42.844 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:42.844 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:42.844 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 72 k 2020-03-09T00:22:42.844 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:42.845 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:42.845 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:42.845 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2020-03-09T00:22:42.845 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:42.845 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 72 k 2020-03-09T00:22:42.845 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 215 k 2020-03-09T00:22:42.846 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:42.904 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:42.909 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:42.975 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-15.1.0-1904.g9eb9490.el8.x86_64.rpm 552 kB/s | 72 kB 00:00 2020-03-09T00:22:42.976 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:42.976 INFO:teuthology.orchestra.run.smithi104.stdout:Total 547 kB/s | 72 kB 00:00 2020-03-09T00:22:42.978 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:42.989 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:42.989 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:43.009 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:43.011 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-03-09T00:22:43.011 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:43.038 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:43.038 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:43.121 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:54 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:43.129 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-fuse 2020-03-09T00:22:43.152 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:43.165 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:43.165 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:43.165 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2020-03-09T00:22:43.165 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:43.166 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:22:43.166 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2020-03-09T00:22:43.300 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:43.451 DEBUG:teuthology.parallel:result is None 2020-03-09T00:22:43.497 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:43.515 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:43.516 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:43.516 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2020-03-09T00:22:43.516 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:43.516 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:43.517 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 21 k 2020-03-09T00:22:43.517 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:43.517 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:43.517 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:43.517 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2020-03-09T00:22:43.517 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:43.518 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 21 k 2020-03-09T00:22:43.518 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 76 k 2020-03-09T00:22:43.518 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:43.532 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:54 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:43.630 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-15.1.0-1904.g9eb9490.el8.x86_64 195 kB/s | 21 kB 00:00 2020-03-09T00:22:43.630 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:43.630 INFO:teuthology.orchestra.run.smithi050.stdout:Total 193 kB/s | 21 kB 00:00 2020-03-09T00:22:43.631 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:43.634 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:43.634 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:43.639 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:43.639 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:43.686 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:43.744 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:43.744 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:43.744 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:43.744 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:43.744 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:43.745 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:43.917 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:43.918 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:43.919 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:43.919 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:43.919 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:43.919 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 72 k 2020-03-09T00:22:43.919 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:43.919 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:43.920 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:43.920 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2020-03-09T00:22:43.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:43.920 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 72 k 2020-03-09T00:22:43.920 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 215 k 2020-03-09T00:22:43.920 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:44.047 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:44.050 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-fuse-15.1.0-1904.g9eb9490.el8.x86_64.rpm 551 kB/s | 72 kB 00:00 2020-03-09T00:22:44.051 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:44.051 INFO:teuthology.orchestra.run.smithi148.stdout:Total 546 kB/s | 72 kB 00:00 2020-03-09T00:22:44.053 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:44.064 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:44.064 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:44.113 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:44.114 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:44.240 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:44.309 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:44.310 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:44.310 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:44.310 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:44.310 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:44.310 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:44.376 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:44.396 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2020-03-09T00:22:44.431 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2020-03-09T00:22:44.599 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:44.805 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:55 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:44.858 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:55 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:44.866 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:44.866 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:44.866 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:44.867 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:44.867 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:44.867 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:45.146 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:45.150 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-mirror 2020-03-09T00:22:45.257 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:45.258 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:45.258 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:45.380 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2020-03-09T00:22:45.475 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:45.476 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:45.476 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2020-03-09T00:22:45.477 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:45.477 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:45.477 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 2.7 M 2020-03-09T00:22:45.477 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:45.477 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:45.477 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:45.478 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2020-03-09T00:22:45.478 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:45.478 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.7 M 2020-03-09T00:22:45.478 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 12 M 2020-03-09T00:22:45.478 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:45.600 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:56 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:45.749 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-15.1.0-1904.g9eb9490.el8.x86_64.rpm 9.9 MB/s | 2.7 MB 00:00 2020-03-09T00:22:45.750 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:45.750 INFO:teuthology.orchestra.run.smithi104.stdout:Total 9.8 MB/s | 2.7 MB 00:00 2020-03-09T00:22:45.752 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:45.765 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:45.765 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:45.810 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:56 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:45.823 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:45.823 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:46.095 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:46.209 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:46.211 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:46.211 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:46.214 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:46.337 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2020-03-09T00:22:46.427 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:46.443 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:46.443 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:46.444 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2020-03-09T00:22:46.444 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:46.444 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:46.444 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 2.7 M 2020-03-09T00:22:46.444 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:46.444 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:46.444 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:46.445 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2020-03-09T00:22:46.445 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:46.445 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 2.7 M 2020-03-09T00:22:46.445 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 12 M 2020-03-09T00:22:46.445 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:46.728 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:46.766 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:57 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:46.882 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-mirror-15.1.0-1904.g9eb9490.el8.x86_64.rpm 6.1 MB/s | 2.7 MB 00:00 2020-03-09T00:22:46.883 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:46.884 INFO:teuthology.orchestra.run.smithi148.stdout:Total 6.1 MB/s | 2.7 MB 00:00 2020-03-09T00:22:46.886 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:46.899 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:46.899 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:46.958 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:46.958 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:46.971 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:46.972 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:46.972 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:46.972 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:46.972 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:46.972 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:47.056 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:47.168 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:47.169 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:47.169 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:47.232 INFO:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2020-03-09T00:22:47.296 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2020-03-09T00:22:47.491 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:47.634 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:58 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:47.694 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:47.724 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:58 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:47.995 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:48.022 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:48.024 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2020-03-09T00:22:48.025 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:48.025 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:48.025 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:48.026 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2020-03-09T00:22:48.026 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:48.026 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:48.026 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2020-03-09T00:22:48.026 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2020-03-09T00:22:48.026 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2020-03-09T00:22:48.027 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:48.027 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 141 k 2020-03-09T00:22:48.027 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 504 k 2020-03-09T00:22:48.027 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2020-03-09T00:22:48.107 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:48.108 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:48.108 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:48.178 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-15.1.0-1904.g9eb9490.el8.x86_64.rpm 940 kB/s | 141 kB 00:00 2020-03-09T00:22:48.178 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:48.178 INFO:teuthology.orchestra.run.smithi104.stdout:Total 934 kB/s | 141 kB 00:00 2020-03-09T00:22:48.179 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2020-03-09T00:22:48.190 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2020-03-09T00:22:48.190 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2020-03-09T00:22:48.238 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2020-03-09T00:22:48.239 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2020-03-09T00:22:48.243 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2020-03-09T00:22:48.280 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:48.281 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:48.281 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:48.281 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:48.281 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:48.281 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:48.343 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2020-03-09T00:22:48.507 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:48.554 INFO:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-nbd 2020-03-09T00:22:48.675 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:59 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:48.740 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:48.954 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:48.961 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:59 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:49.004 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:49.005 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:49.005 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2020-03-09T00:22:49.005 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:49.005 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:22:49.005 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2020-03-09T00:22:49.064 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:49.065 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:49.065 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:49.210 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2020-03-09T00:22:49.291 DEBUG:teuthology.parallel:result is None 2020-03-09T00:22:49.343 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-03-09T00:22:49.344 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:49.344 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:49.344 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2020-03-09T00:22:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:49.345 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2020-03-09T00:22:49.345 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2020-03-09T00:22:49.345 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2020-03-09T00:22:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:49.345 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 141 k 2020-03-09T00:22:49.346 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 504 k 2020-03-09T00:22:49.346 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2020-03-09T00:22:49.495 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-nbd-15.1.0-1904.g9eb9490.el8.x86_64.rpm 936 kB/s | 141 kB 00:00 2020-03-09T00:22:49.496 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:49.496 INFO:teuthology.orchestra.run.smithi148.stdout:Total 929 kB/s | 141 kB 00:00 2020-03-09T00:22:49.498 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2020-03-09T00:22:49.510 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2020-03-09T00:22:49.510 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2020-03-09T00:22:49.559 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2020-03-09T00:22:49.559 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2020-03-09T00:22:49.638 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:00 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:49.665 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2020-03-09T00:22:49.833 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:49.923 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:15.1.0-1904.g9eb9490.el8.x86_64 is already installed. 2020-03-09T00:22:50.036 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:50.037 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2020-03-09T00:22:50.037 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:50.039 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:50.170 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2020-03-09T00:22:50.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:50.300 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:50.300 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2020-03-09T00:22:50.300 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:50.300 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:22:50.300 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-03-09T00:22:50.570 DEBUG:teuthology.parallel:result is None 2020-03-09T00:22:50.598 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:01 ago on Mon 09 Mar 2020 12:20:49 AM UTC. 2020-03-09T00:22:51.158 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:51.158 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:51.158 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:51.159 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 72 k 2020-03-09T00:22:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:51.159 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:51.159 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:51.159 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2020-03-09T00:22:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:51.160 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 72 k 2020-03-09T00:22:51.160 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 215 k 2020-03-09T00:22:51.160 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:51.291 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-15.1.0-1904.g9eb9490.el8.x86_64.rpm 547 kB/s | 72 kB 00:00 2020-03-09T00:22:51.292 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:51.292 INFO:teuthology.orchestra.run.smithi050.stdout:Total 542 kB/s | 72 kB 00:00 2020-03-09T00:22:51.294 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:51.306 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:51.306 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:51.358 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:51.358 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:51.485 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:51.628 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:51.845 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:52.089 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:52.090 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:52.090 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:52.090 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:52.090 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:52.378 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2020-03-09T00:22:52.628 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for process with pid 24625 to finish. 2020-03-09T00:22:53.831 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:01 ago on Mon 09 Mar 2020 12:22:52 AM UTC. 2020-03-09T00:22:54.214 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:54.215 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:54.215 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2020-03-09T00:22:54.215 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:54.215 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:54.215 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 2.7 M 2020-03-09T00:22:54.216 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:54.216 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:54.216 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:54.216 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2020-03-09T00:22:54.216 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:54.216 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.7 M 2020-03-09T00:22:54.217 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 12 M 2020-03-09T00:22:54.217 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:54.636 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-15.1.0-1904.g9eb9490.el8.x86_64.rpm 6.4 MB/s | 2.7 MB 00:00 2020-03-09T00:22:54.636 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:54.637 INFO:teuthology.orchestra.run.smithi050.stdout:Total 6.4 MB/s | 2.7 MB 00:00 2020-03-09T00:22:54.639 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:54.653 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:54.653 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:54.715 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:54.715 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:55.091 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:55.624 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:56.003 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:56.003 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:56.003 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:56.004 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:56.004 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:56.004 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:57.554 INFO:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2020-03-09T00:22:57.990 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:05 ago on Mon 09 Mar 2020 12:22:52 AM UTC. 2020-03-09T00:22:58.371 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2020-03-09T00:22:58.372 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:58.372 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2020-03-09T00:22:58.372 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:58.372 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2020-03-09T00:22:58.372 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:15.1.0-1904.g9eb9490.el8 ceph 141 k 2020-03-09T00:22:58.372 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2020-03-09T00:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2020-03-09T00:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2020-03-09T00:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 141 k 2020-03-09T00:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 504 k 2020-03-09T00:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2020-03-09T00:22:58.524 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-15.1.0-1904.g9eb9490.el8.x86_64.rpm 933 kB/s | 141 kB 00:00 2020-03-09T00:22:58.524 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2020-03-09T00:22:58.525 INFO:teuthology.orchestra.run.smithi050.stdout:Total 926 kB/s | 141 kB 00:00 2020-03-09T00:22:58.526 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2020-03-09T00:22:58.539 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2020-03-09T00:22:58.539 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2020-03-09T00:22:58.591 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2020-03-09T00:22:58.592 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2020-03-09T00:22:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2020-03-09T00:22:58.926 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:59.140 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 1/1 2020-03-09T00:22:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:59.412 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2020-03-09T00:22:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:15.1.0-1904.g9eb9490.el8.x86_64 2020-03-09T00:22:59.412 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:22:59.413 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2020-03-09T00:22:59.755 DEBUG:teuthology.parallel:result is None 2020-03-09T00:22:59.755 INFO:teuthology.orchestra.run.smithi148:> sudo lsb_release -is 2020-03-09T00:22:59.800 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS 2020-03-09T00:22:59.801 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:22:59.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:22:59.956 INFO:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-09T00:22:59.978 INFO:teuthology.orchestra.run.smithi148.stdout:15.1.0-1904.g9eb9490.el8 2020-03-09T00:22:59.978 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1904.g9eb9490.el8 2020-03-09T00:22:59.978 INFO:teuthology.task.install:The correct ceph version 15.1.0-1904.g9eb9490 is installed. 2020-03-09T00:22:59.979 INFO:teuthology.orchestra.run.smithi104:> sudo lsb_release -is 2020-03-09T00:23:00.022 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS 2020-03-09T00:23:00.024 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:23:00.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:23:00.191 INFO:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-09T00:23:00.212 INFO:teuthology.orchestra.run.smithi104.stdout:15.1.0-1904.g9eb9490.el8 2020-03-09T00:23:00.212 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1904.g9eb9490.el8 2020-03-09T00:23:00.212 INFO:teuthology.task.install:The correct ceph version 15.1.0-1904.g9eb9490 is installed. 2020-03-09T00:23:00.212 INFO:teuthology.orchestra.run.smithi050:> sudo lsb_release -is 2020-03-09T00:23:00.260 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS 2020-03-09T00:23:00.261 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:23:00.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:23:00.420 INFO:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-09T00:23:00.444 INFO:teuthology.orchestra.run.smithi050.stdout:15.1.0-1904.g9eb9490.el8 2020-03-09T00:23:00.444 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1904.g9eb9490.el8 2020-03-09T00:23:00.445 INFO:teuthology.task.install:The correct ceph version 15.1.0-1904.g9eb9490 is installed. 2020-03-09T00:23:00.445 INFO:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2020-03-09T00:23:00.489 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS 2020-03-09T00:23:00.490 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-09T00:23:00.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:23:00.661 INFO:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-09T00:23:00.682 INFO:teuthology.orchestra.run.smithi139.stdout:15.1.0-1904.g9eb9490.el8 2020-03-09T00:23:00.683 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1904.g9eb9490.el8 2020-03-09T00:23:00.683 INFO:teuthology.task.install:The correct ceph version 15.1.0-1904.g9eb9490 is installed. 2020-03-09T00:23:00.683 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-09T00:23:00.685 INFO:teuthology.orchestra.run.smithi148:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-09T00:23:00.718 INFO:teuthology.orchestra.run.smithi104:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-09T00:23:00.759 INFO:teuthology.orchestra.run.smithi050:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-09T00:23:00.819 INFO:teuthology.orchestra.run.smithi139:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-09T00:23:00.852 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-09T00:23:00.852 INFO:teuthology.orchestra.run.smithi148:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-09T00:23:00.884 INFO:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-09T00:23:00.955 INFO:teuthology.orchestra.run.smithi104:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-09T00:23:00.987 INFO:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-09T00:23:01.058 INFO:teuthology.orchestra.run.smithi050:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-09T00:23:01.096 INFO:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-09T00:23:01.170 INFO:teuthology.orchestra.run.smithi139:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-09T00:23:01.203 INFO:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-09T00:23:01.273 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-09T00:23:01.273 INFO:teuthology.orchestra.run.smithi148:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-09T00:23:01.320 INFO:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-09T00:23:01.390 INFO:teuthology.orchestra.run.smithi104:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-09T00:23:01.436 INFO:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-09T00:23:01.508 INFO:teuthology.orchestra.run.smithi050:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-09T00:23:01.554 INFO:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-09T00:23:01.629 INFO:teuthology.orchestra.run.smithi139:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-09T00:23:01.675 INFO:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-09T00:23:01.745 INFO:teuthology.run_tasks:Running task ceph... 2020-03-09T00:23:01.842 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-03-09T00:23:01.842 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2020-03-09T00:23:01.845 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2020-03-09T00:23:01.848 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 777 /var/log/ceph 2020-03-09T00:23:01.850 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 777 /var/log/ceph 2020-03-09T00:23:01.881 INFO:tasks.ceph:Disabling ceph logrotate... 2020-03-09T00:23:01.882 INFO:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2020-03-09T00:23:01.920 INFO:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2020-03-09T00:23:01.922 INFO:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/logrotate.d/ceph 2020-03-09T00:23:01.925 INFO:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/logrotate.d/ceph 2020-03-09T00:23:01.955 INFO:tasks.ceph:Creating extra log directories... 2020-03-09T00:23:01.955 INFO:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-03-09T00:23:01.994 INFO:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-03-09T00:23:01.996 INFO:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-03-09T00:23:01.999 INFO:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-03-09T00:23:02.029 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-03-09T00:23:02.029 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'global': {'ms tcp read timeout': 5, 'ms inject socket failures': 2500, 'mon client directed command retry': 5, 'osd_async_recovery_min_cost': 1}, 'osd': {'debug journal': 20, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'osd debug verify missing on start': True, 'debug ms': 20, 'osd shutdown pgref assert': True, 'debug monc': 20, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd scrub min interval': 60, 'osd failsafe full ratio': 0.95, 'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'osd scrub max interval': 120, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug reject backfill probability': 0.3, 'debug rocksdb': 10, 'osd debug verify cached snaps': True, 'osd max markdown count': 1000, 'osd max backfills': 3, 'bluestore block size': 96636764160, 'debug filestore': 20}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'debug ms': 1}}} 2020-03-09T00:23:02.029 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (4, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'ceph', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'fs': 'xfs', 'conf': {'global': {'ms tcp read timeout': 5, 'ms inject socket failures': 2500, 'mon client directed command retry': 5, 'osd_async_recovery_min_cost': 1}, 'osd': {'debug journal': 20, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'osd debug verify missing on start': True, 'debug ms': 20, 'osd shutdown pgref assert': True, 'debug monc': 20, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd scrub min interval': 60, 'osd failsafe full ratio': 0.95, 'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'osd scrub max interval': 120, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug reject backfill probability': 0.3, 'debug rocksdb': 10, 'osd debug verify cached snaps': True, 'osd max markdown count': 1000, 'osd max backfills': 3, 'bluestore block size': 96636764160, 'debug filestore': 20}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'debug ms': 1}}, 'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b'}}, {'thrashosds': {'chance_pgpnum_fix': 1, 'aggressive_pg_num_changes': False, 'min_in': 8, 'chance_pgnum_grow': 1, 'timeout': 1200, 'chance_pgnum_shrink': 1}}, {'rados': {'op_weights': {'snap_remove': 50, 'snap_create': 50, 'rollback': 50, 'setattr': 25, 'read': 100, 'copy_from': 50, 'write': 0, 'rmattr': 25, 'append': 100, 'delete': 50}, 'objects': 50, 'erasure_code_profile': {'c': 2, 'name': 'shecprofile', 'plugin': 'shec', 'crush-failure-domain': 'osd', 'k': 4, 'm': 3}, 'ops': 400, 'ec_pool': True, 'clients': ['client.0'], 'write_append_excl': False}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.27946', 'owner': 'scheduled_sage@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'targets': {'smithi148.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRHf68nfkBAqbAK/zpskKJkcMrbu/r9PJWZKqFOBYfsIjDWlvJ/OqIQF4gLN243MMNZWy+deax+JAh5664fWWO0zcp60EQbjWNAj60/p7ry1CqwJjmS9Y/V7luxfYTtHXLof5/cJNrB+829juP+3OKwhqSuZejN6+0xF/fUitPiPaxRvZPEp2iTYLHqJ0ecJ5RrBcr/xJ+fnjrhwP95AvwukKSz1QwPEbmiZ+BI/wo4oJIybRqckzhzBzBGMVOcOij1xQYudCVRHk0X/45Y127veGbSDQ/IvT5b4Vr1Hagv5MNn4N/SlQbz+EoCCb8gS4Vr7bjLdpsMV5hJvYgN+Zy3J64HrdFSLlDp5uVkWaI0UQge3P80z1dd4VRE86LTfssAUovWEOE1jkixIT609V6GxeSFm7OacLEIEIM+m1cqVDsjsGdIjGFQ5fKHnLoNPYhqQMPKptqc7+DWIRpn9KYw5PO2aGNxGRygvxjFSYAAUhszQ1+TFr/7iBs7/U2CKE=', 'smithi050.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDalU+i1g46Gk8/fAPywT5D5W1tLlQMV7TqFtSnJ5N9sdptiKCuJ+hco8RcZUgxZDlurA79xmu952hQiQX4goNPOHDf4nJl+67o/esQFBrAKVHCGzjBrslFUNHYYpghL6wi3/3VEFBAhe5OEC4G8VNTCepruQVlkEb0MAe4g71AaTQmmtijLPIfn+HbBHhuKMmIU2t0yJUYfBVlKBlgbme1FzSF/JLQMgAgeUMz0OQG0EW6REJ/w1Gcjly6FrYCeAeUKV4FH1PP0ADBp4gqvrEuMFDyvjbtHwy3R5EJVbSPEMXJDpZCAO4vJP7fcqsNRzJNQcCg92LrQagC55fEJ/29a2XgkDfbiURtZmJ72bvD906Gcz6TpHFPtgpdRNWJuP9Z3jy2zW2w1ZFiwn7FktTvZS13JLt2wrqJTDYfR1JoqplzwzCtrQKmjnx2NzA2RwfP2hSxx8mtQpiqLpmaFZD77gxSgVL4OQq7aeKVcp0yKJAOsJv++IgRX3Qw6Ob+WS8=', 'smithi139.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMous9Ne3mtgYoORWx5JJz6ldzqlcHZPKlsZA5t6Hc0q3fjKCFLVGxaCMYjxNQWKVh/i2dgi1ttxZxmco78IZR3vVcFUG6CoWQ+a+o8qvQukOFoMhVOr0wop93wvd/yhf00DIqk/4SPFwlMMpX/LmuXHSMSpwrYzw02DVNTDMaTNtBfsLXSi1BQWxwFh7PZ1VrRix16j/JP8oDbj8BKaA5cf83k1G7d62i5DpaD4eG0pdsFUMnFqZj/m1st54CyV3FlivTgjh0zZtz3rcIi1EZtrXhPW9B9VwtkrBTv4XgEROwr3L7lgXOnX+Gsgu82XIuqLXuungrjZo54QPBbp2WsRPDMGTNAat2FRpWCPcRbic3jmgzakDrA0UptizqjAIh2v4T/htnRft+CqNLiq8AeftpZKcIdVxWCIAxBZpuPhuEOAMqfrna1VIGUPrsbpBz1WlXF075PxlsGuXT3FYvEHwLsks0RyS6xTBEdmWZxmdpyrbEsZdN1FQGXyhn8FM=', 'smithi104.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZGIxAw51Rgx3n37R+EOgbt3eHDwLITnZc3qMfay4K3J+yJpAGW5+X2kD03heXhxobOTq908Boo8CL26wvNDSsJIgOoXhBjiRRJHlRKcebsw6gglsAfAh2njV9CrkcSd5wWYDR5JtPzNAx5c4EDBvRFZm/6VjDU0kksc60UaxBNVgQ92XZLczP3Q92b/XcaEKcsN1Mn+aj3L93pvUZGFdw1JhCLRBtj9H/tKRSVBYpygZIx7TVZlHheqk/SsrtcgoO6kshZfv5Ouc+ZVruv9B17yDruOpWWZlgBeh1gF8fE0l08nzbjlTkm+KVg7RW6pop1U1bj77t2vleKDbCJlL8JpmHWk2R5uBDhGU/RJhN2y0FQJpSaCjCzZORFXGL0DoPqvbV2vh4Eh5v5Ysf5LjmjzPCFCaDfQV1qRtLVzPIkq+dwYy/myQmy9gArGmjDGaHwDqTlCNivCmdCXuby4jhmplipZL29MlsiD/qfksWzcVAaMi0NtKhiL1v5zyBRQM='}, 'job_id': '4838152', 'tube': 'smithi', 'suite_branch': 'wip-sage4-testing-2020-03-08-0938', 'priority': 90, 'os_version': '8.1', 'first_in_suite': False, 'branch': 'wip-sage4-testing-2020-03-08-0938', 'suite': 'rados', 'email': None, 'archive_path': '/home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838152', 'description': 'rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-comp-zlib.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_8.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'name': 'sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'os_type': 'centos', 'overrides': {'ceph': {'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'conf': {'global': {'ms tcp read timeout': 5, 'ms inject socket failures': 2500, 'mon client directed command retry': 5, 'osd_async_recovery_min_cost': 1}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'mon osd backfillfull_ratio': 0.85, 'debug journal': 20, 'osd scrub max interval': 120, 'bluestore compression algorithm': 'zlib', 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug verify missing on start': True, 'debug ms': 20, 'osd debug reject backfill probability': 0.3, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'bluestore fsck on mount': True, 'osd max backfills': 3, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd debug verify cached snaps': True, 'osd max markdown count': 1000, 'osd op queue': 'debug_random', 'bluestore block size': 96636764160, 'osd scrub min interval': 60, 'debug filestore': 20, 'osd failsafe full ratio': 0.95}}, 'fs': 'xfs', 'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b'}}, 'admin_socket': {'branch': 'wip-sage4-testing-2020-03-08-0938'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'branch': 'wip-sage4-testing-2020-03-08-0938'}}} 2020-03-09T00:23:02.030 INFO:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-03-09T00:23:02.067 INFO:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-03-09T00:23:02.069 INFO:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-03-09T00:23:02.071 INFO:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-03-09T00:23:02.086 INFO:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2020-03-09T00:23:02.121 INFO:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2020-03-09T00:23:02.124 INFO:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/run/ceph 2020-03-09T00:23:02.127 INFO:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2020-03-09T00:23:02.211 DEBUG:teuthology.orchestra.remote:smithi139:/scratch_devs is 72B 2020-03-09T00:23:02.260 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:02.261 INFO:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-03-09T00:23:02.275 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-03-09T00:23:02.276 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.276 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 94213 Links: 1 2020-03-09T00:23:02.276 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.276 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.277 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2020-03-09 00:22:40.666130947 +0000 2020-03-09T00:23:02.277 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2020-03-09 00:20:27.704699579 +0000 2020-03-09T00:23:02.277 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2020-03-09 00:20:27.704699579 +0000 2020-03-09T00:23:02.277 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2020-03-09T00:23:02.289 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2020-03-09T00:23:02.289 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2020-03-09T00:23:02.290 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000216268 s, 2.4 MB/s 2020-03-09T00:23:02.293 INFO:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-03-09T00:23:02.348 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-03-09T00:23:02.348 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.348 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 88967 Links: 1 2020-03-09T00:23:02.348 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.348 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.349 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2020-03-09 00:22:40.666130947 +0000 2020-03-09T00:23:02.349 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2020-03-09 00:20:28.043692228 +0000 2020-03-09T00:23:02.349 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2020-03-09 00:20:28.043692228 +0000 2020-03-09T00:23:02.349 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2020-03-09T00:23:02.361 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2020-03-09T00:23:02.362 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2020-03-09T00:23:02.362 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000187145 s, 2.7 MB/s 2020-03-09T00:23:02.364 INFO:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-03-09T00:23:02.418 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-03-09T00:23:02.419 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.419 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 94277 Links: 1 2020-03-09T00:23:02.419 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.419 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.419 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2020-03-09 00:22:40.666130947 +0000 2020-03-09T00:23:02.419 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2020-03-09 00:20:28.425683945 +0000 2020-03-09T00:23:02.420 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2020-03-09 00:20:28.425683945 +0000 2020-03-09T00:23:02.420 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2020-03-09T00:23:02.432 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2020-03-09T00:23:02.432 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2020-03-09T00:23:02.432 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000213025 s, 2.4 MB/s 2020-03-09T00:23:02.434 INFO:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-03-09T00:23:02.488 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-03-09T00:23:02.488 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.488 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 93463 Links: 1 2020-03-09T00:23:02.489 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.489 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.489 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2020-03-09 00:22:40.666130947 +0000 2020-03-09T00:23:02.489 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2020-03-09 00:20:28.785676139 +0000 2020-03-09T00:23:02.489 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2020-03-09 00:20:28.785676139 +0000 2020-03-09T00:23:02.489 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2020-03-09T00:23:02.501 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2020-03-09T00:23:02.501 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2020-03-09T00:23:02.502 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000204928 s, 2.5 MB/s 2020-03-09T00:23:02.504 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-03-09T00:23:02.504 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:02.504 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-03-09T00:23:02.505 INFO:tasks.ceph:dev map: {'osd.10': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_4', 'osd.14': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:02.524 DEBUG:teuthology.orchestra.remote:smithi148:/scratch_devs is 72B 2020-03-09T00:23:02.573 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:02.573 INFO:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-03-09T00:23:02.589 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-03-09T00:23:02.589 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.589 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 88770 Links: 1 2020-03-09T00:23:02.589 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.590 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.590 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2020-03-09 00:22:47.855454647 +0000 2020-03-09T00:23:02.590 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2020-03-09 00:20:26.177969847 +0000 2020-03-09T00:23:02.590 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2020-03-09 00:20:26.177969847 +0000 2020-03-09T00:23:02.590 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2020-03-09T00:23:02.603 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2020-03-09T00:23:02.603 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2020-03-09T00:23:02.604 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000229084 s, 2.2 MB/s 2020-03-09T00:23:02.607 INFO:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-03-09T00:23:02.663 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-03-09T00:23:02.663 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.663 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 95305 Links: 1 2020-03-09T00:23:02.663 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.663 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.664 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2020-03-09 00:22:47.855454647 +0000 2020-03-09T00:23:02.664 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2020-03-09 00:20:26.512961678 +0000 2020-03-09T00:23:02.664 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2020-03-09 00:20:26.512961678 +0000 2020-03-09T00:23:02.664 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2020-03-09T00:23:02.677 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2020-03-09T00:23:02.677 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2020-03-09T00:23:02.677 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.00018461 s, 2.8 MB/s 2020-03-09T00:23:02.680 INFO:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-03-09T00:23:02.735 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-03-09T00:23:02.736 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.736 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 91728 Links: 1 2020-03-09T00:23:02.736 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.736 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.736 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2020-03-09 00:22:47.856454623 +0000 2020-03-09T00:23:02.736 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2020-03-09 00:20:26.848953486 +0000 2020-03-09T00:23:02.736 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2020-03-09 00:20:26.848953486 +0000 2020-03-09T00:23:02.737 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2020-03-09T00:23:02.749 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2020-03-09T00:23:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2020-03-09T00:23:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000193973 s, 2.6 MB/s 2020-03-09T00:23:02.752 INFO:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-03-09T00:23:02.806 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-03-09T00:23:02.807 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.807 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 6h/6d Inode: 95368 Links: 1 2020-03-09T00:23:02.807 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.807 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.807 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2020-03-09 00:22:47.856454623 +0000 2020-03-09T00:23:02.807 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2020-03-09 00:20:27.197944976 +0000 2020-03-09T00:23:02.808 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2020-03-09 00:20:27.197944976 +0000 2020-03-09T00:23:02.808 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: - 2020-03-09T00:23:02.820 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2020-03-09T00:23:02.820 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2020-03-09T00:23:02.820 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000224432 s, 2.3 MB/s 2020-03-09T00:23:02.823 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-03-09T00:23:02.823 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:02.823 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-03-09T00:23:02.823 INFO:tasks.ceph:dev map: {'osd.0': '/dev/vg_nvme/lv_4', 'osd.12': '/dev/vg_nvme/lv_3', 'osd.4': '/dev/vg_nvme/lv_1', 'osd.8': '/dev/vg_nvme/lv_2'} 2020-03-09T00:23:02.846 DEBUG:teuthology.orchestra.remote:smithi050:/scratch_devs is 72B 2020-03-09T00:23:02.896 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:02.896 INFO:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-03-09T00:23:02.913 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-03-09T00:23:02.913 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.913 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 98705 Links: 1 2020-03-09T00:23:02.913 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.914 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.914 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2020-03-09 00:22:55.450451206 +0000 2020-03-09T00:23:02.914 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2020-03-09 00:20:24.563044570 +0000 2020-03-09T00:23:02.914 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2020-03-09 00:20:24.563044570 +0000 2020-03-09T00:23:02.914 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2020-03-09T00:23:02.931 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2020-03-09T00:23:02.931 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2020-03-09T00:23:02.931 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000113708 s, 4.5 MB/s 2020-03-09T00:23:02.934 INFO:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-03-09T00:23:02.991 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-03-09T00:23:02.991 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:02.991 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 99506 Links: 1 2020-03-09T00:23:02.991 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:02.991 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:02.991 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2020-03-09 00:22:55.450451206 +0000 2020-03-09T00:23:02.992 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2020-03-09 00:20:24.910035789 +0000 2020-03-09T00:23:02.992 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2020-03-09 00:20:24.910035789 +0000 2020-03-09T00:23:02.992 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2020-03-09T00:23:03.008 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2020-03-09T00:23:03.008 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2020-03-09T00:23:03.008 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000116873 s, 4.4 MB/s 2020-03-09T00:23:03.011 INFO:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-03-09T00:23:03.067 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-03-09T00:23:03.067 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:03.067 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 93183 Links: 1 2020-03-09T00:23:03.067 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:03.067 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:03.068 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2020-03-09 00:22:55.450451206 +0000 2020-03-09T00:23:03.068 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2020-03-09 00:20:25.293026098 +0000 2020-03-09T00:23:03.068 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2020-03-09 00:20:25.293026098 +0000 2020-03-09T00:23:03.068 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2020-03-09T00:23:03.084 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2020-03-09T00:23:03.084 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2020-03-09T00:23:03.084 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000104883 s, 4.9 MB/s 2020-03-09T00:23:03.087 INFO:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-03-09T00:23:03.142 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-03-09T00:23:03.142 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:03.143 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 95753 Links: 1 2020-03-09T00:23:03.143 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:03.143 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:03.143 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2020-03-09 00:22:55.450451206 +0000 2020-03-09T00:23:03.143 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2020-03-09 00:20:25.660016811 +0000 2020-03-09T00:23:03.143 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2020-03-09 00:20:25.660016811 +0000 2020-03-09T00:23:03.144 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2020-03-09T00:23:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2020-03-09T00:23:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2020-03-09T00:23:03.159 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.456e-05 s, 6.1 MB/s 2020-03-09T00:23:03.162 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-03-09T00:23:03.162 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:03.162 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-03-09T00:23:03.162 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_4', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.13': '/dev/vg_nvme/lv_3', 'osd.5': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:03.184 DEBUG:teuthology.orchestra.remote:smithi104:/scratch_devs is 72B 2020-03-09T00:23:03.232 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:03.232 INFO:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-03-09T00:23:03.248 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-03-09T00:23:03.249 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:03.249 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 94039 Links: 1 2020-03-09T00:23:03.249 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:03.249 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:03.249 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2020-03-09 00:22:46.566411617 +0000 2020-03-09T00:23:03.249 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2020-03-09 00:20:24.560851581 +0000 2020-03-09T00:23:03.249 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2020-03-09 00:20:24.560851581 +0000 2020-03-09T00:23:03.250 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2020-03-09T00:23:03.263 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2020-03-09T00:23:03.263 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2020-03-09T00:23:03.264 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000205776 s, 2.5 MB/s 2020-03-09T00:23:03.266 INFO:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-03-09T00:23:03.322 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-03-09T00:23:03.322 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:03.322 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 94802 Links: 1 2020-03-09T00:23:03.322 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:03.322 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:03.323 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2020-03-09 00:22:46.566411617 +0000 2020-03-09T00:23:03.323 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2020-03-09 00:20:24.942846907 +0000 2020-03-09T00:23:03.323 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2020-03-09 00:20:24.942846907 +0000 2020-03-09T00:23:03.323 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2020-03-09T00:23:03.336 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2020-03-09T00:23:03.336 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2020-03-09T00:23:03.336 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000188231 s, 2.7 MB/s 2020-03-09T00:23:03.339 INFO:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-03-09T00:23:03.394 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-03-09T00:23:03.394 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:03.394 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 95945 Links: 1 2020-03-09T00:23:03.394 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:03.394 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:03.394 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2020-03-09 00:22:46.566411617 +0000 2020-03-09T00:23:03.394 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2020-03-09 00:20:25.287842687 +0000 2020-03-09T00:23:03.395 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2020-03-09 00:20:25.287842687 +0000 2020-03-09T00:23:03.395 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2020-03-09T00:23:03.407 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2020-03-09T00:23:03.407 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2020-03-09T00:23:03.407 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000211147 s, 2.4 MB/s 2020-03-09T00:23:03.410 INFO:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-03-09T00:23:03.465 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-03-09T00:23:03.465 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-09T00:23:03.465 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 97787 Links: 1 2020-03-09T00:23:03.465 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-09T00:23:03.466 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2020-03-09T00:23:03.466 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2020-03-09 00:22:46.566411617 +0000 2020-03-09T00:23:03.466 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2020-03-09 00:20:25.639838381 +0000 2020-03-09T00:23:03.466 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2020-03-09 00:20:25.639838381 +0000 2020-03-09T00:23:03.466 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2020-03-09T00:23:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2020-03-09T00:23:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2020-03-09T00:23:03.479 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000225133 s, 2.3 MB/s 2020-03-09T00:23:03.481 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-03-09T00:23:03.481 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-09T00:23:03.481 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-03-09T00:23:03.482 INFO:tasks.ceph:dev map: {'osd.11': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_4', 'osd.15': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:03.482 INFO:tasks.ceph:Generating config... 2020-03-09T00:23:03.485 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2020-03-09T00:23:03.485 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2020-03-09T00:23:03.485 INFO:tasks.ceph:[global] mon client directed command retry = 5 2020-03-09T00:23:03.486 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2020-03-09T00:23:03.486 INFO:tasks.ceph:[osd] debug journal = 20 2020-03-09T00:23:03.486 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2020-03-09T00:23:03.486 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2020-03-09T00:23:03.486 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2020-03-09T00:23:03.486 INFO:tasks.ceph:[osd] debug ms = 20 2020-03-09T00:23:03.486 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2020-03-09T00:23:03.486 INFO:tasks.ceph:[osd] debug monc = 20 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] osd op queue = debug_random 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2020-03-09T00:23:03.487 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] debug osd = 25 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] debug bluestore = 20 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] debug bluefs = 20 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] debug rocksdb = 10 2020-03-09T00:23:03.488 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2020-03-09T00:23:03.489 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2020-03-09T00:23:03.489 INFO:tasks.ceph:[osd] osd max backfills = 3 2020-03-09T00:23:03.489 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2020-03-09T00:23:03.489 INFO:tasks.ceph:[osd] debug filestore = 20 2020-03-09T00:23:03.489 INFO:tasks.ceph:[mon] debug mon = 20 2020-03-09T00:23:03.489 INFO:tasks.ceph:[mon] debug paxos = 20 2020-03-09T00:23:03.489 INFO:tasks.ceph:[mon] debug ms = 1 2020-03-09T00:23:03.490 INFO:tasks.ceph:Setting up mon.a... 2020-03-09T00:23:03.490 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-03-09T00:23:03.569 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/ceph.keyring 2020-03-09T00:23:03.571 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-03-09T00:23:03.616 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-03-09T00:23:03.683 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.148'), ('mon.c', '172.21.15.139'), ('mon.b', '172.21.15.50')] 2020-03-09T00:23:03.684 INFO:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.148 --add c 172.21.15.139 --add b 172.21.15.50 --print /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:03.761 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:03.761 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: generated fsid 036b76b6-511e-4ec6-9669-0a47550bd261 2020-03-09T00:23:03.761 INFO:teuthology.orchestra.run.smithi148.stdout:epoch 0 2020-03-09T00:23:03.761 INFO:teuthology.orchestra.run.smithi148.stdout:fsid 036b76b6-511e-4ec6-9669-0a47550bd261 2020-03-09T00:23:03.761 INFO:teuthology.orchestra.run.smithi148.stdout:last_changed 2020-03-09T00:23:03.760213+0000 2020-03-09T00:23:03.761 INFO:teuthology.orchestra.run.smithi148.stdout:created 2020-03-09T00:23:03.760213+0000 2020-03-09T00:23:03.762 INFO:teuthology.orchestra.run.smithi148.stdout:min_mon_release 0 (unknown) 2020-03-09T00:23:03.762 INFO:teuthology.orchestra.run.smithi148.stdout:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2020-03-09T00:23:03.762 INFO:teuthology.orchestra.run.smithi148.stdout:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2020-03-09T00:23:03.762 INFO:teuthology.orchestra.run.smithi148.stdout:2: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.c 2020-03-09T00:23:03.762 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-03-09T00:23:03.763 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 036b76b6-511e-4ec6-9669-0a47550bd261... 2020-03-09T00:23:03.764 INFO:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-03-09T00:23:03.766 INFO:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-03-09T00:23:03.768 INFO:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-03-09T00:23:03.771 INFO:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-03-09T00:23:03.820 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2020-03-09T00:23:03.820 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2020-03-09T00:23:03.820 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-03-09T00:23:03.820 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2020-03-09T00:23:03.820 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.821 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2020-03-09T00:23:03.821 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.821 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2020-03-09T00:23:03.821 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.821 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2020-03-09T00:23:03.821 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2020-03-09T00:23:03.821 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.822 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2020-03-09T00:23:03.822 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2020-03-09T00:23:03.822 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.822 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2020-03-09T00:23:03.822 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2020-03-09T00:23:03.822 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.822 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2020-03-09T00:23:03.823 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-03-09T00:23:03.824 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.825 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2020-03-09T00:23:03.825 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.825 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2020-03-09T00:23:03.825 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.825 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2020-03-09T00:23:03.825 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2020-03-09T00:23:03.825 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "172.21.15.148,172.21.15.139,172.21.15.50" 2020-03-09T00:23:03.826 INFO:teuthology.orchestra.run.smithi104.stdout: ms tcp read timeout = 5 2020-03-09T00:23:03.826 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject socket failures = 2500 2020-03-09T00:23:03.826 INFO:teuthology.orchestra.run.smithi104.stdout: mon client directed command retry = 5 2020-03-09T00:23:03.826 INFO:teuthology.orchestra.run.smithi104.stdout: osd_async_recovery_min_cost = 1 2020-03-09T00:23:03.826 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = 036b76b6-511e-4ec6-9669-0a47550bd261 2020-03-09T00:23:03.826 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 120 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2020-03-09T00:23:03.827 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2020-03-09T00:23:03.828 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.828 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2020-03-09T00:23:03.828 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2020-03-09T00:23:03.828 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2020-03-09T00:23:03.828 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.828 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2020-03-09T00:23:03.828 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.829 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: debug journal = 20 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression algorithm = zlib 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression mode = aggressive 2020-03-09T00:23:03.830 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify missing on start = True 2020-03-09T00:23:03.831 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 20 2020-03-09T00:23:03.831 INFO:teuthology.orchestra.run.smithi104.stdout: osd shutdown pgref assert = True 2020-03-09T00:23:03.831 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2020-03-09T00:23:03.831 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd nearfull ratio = 0.8 2020-03-09T00:23:03.831 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = bluestore 2020-03-09T00:23:03.831 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue = debug_random 2020-03-09T00:23:03.831 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub min interval = 60 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: osd failsafe full ratio = 0.95 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd full ratio = 0.9 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue cut off = debug_random 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd backfillfull_ratio = 0.85 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore fsck on mount = True 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 25 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluestore = 20 2020-03-09T00:23:03.832 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluefs = 20 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug reject backfill probability = 0.3 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: debug rocksdb = 10 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify cached snaps = True 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: osd max markdown count = 1000 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: osd max backfills = 3 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore block size = 96636764160 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: debug filestore = 20 2020-03-09T00:23:03.833 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.834 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2020-03-09T00:23:03.834 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2020-03-09T00:23:03.834 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2020-03-09T00:23:03.834 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2020-03-09T00:23:03.834 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2020-03-09T00:23:03.834 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2020-03-09T00:23:03.835 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2020-03-09T00:23:03.836 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2020-03-09T00:23:03.836 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2020-03-09T00:23:03.836 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.836 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2020-03-09T00:23:03.836 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2020-03-09T00:23:03.836 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout: 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-03-09T00:23:03.837 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2020-03-09T00:23:03.838 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2020-03-09T00:23:03.838 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2020-03-09T00:23:03.839 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2020-03-09T00:23:03.839 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2020-03-09T00:23:03.839 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-03-09T00:23:03.839 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2020-03-09T00:23:03.839 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.840 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2020-03-09T00:23:03.841 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2020-03-09T00:23:03.842 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2020-03-09T00:23:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2020-03-09T00:23:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-03-09T00:23:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2020-03-09T00:23:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-03-09T00:23:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.148,172.21.15.139,172.21.15.50" 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: ms tcp read timeout = 5 2020-03-09T00:23:03.844 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 2500 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: osd_async_recovery_min_cost = 1 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 036b76b6-511e-4ec6-9669-0a47550bd261 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.845 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2020-03-09T00:23:03.846 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 120 2020-03-09T00:23:03.846 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.846 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2020-03-09T00:23:03.846 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2020-03-09T00:23:03.846 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.846 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2020-03-09T00:23:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2020-03-09T00:23:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2020-03-09T00:23:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2020-03-09T00:23:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2020-03-09T00:23:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.847 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2020-03-09T00:23:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2020-03-09T00:23:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2020-03-09T00:23:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2020-03-09T00:23:03.848 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2020-03-09T00:23:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2020-03-09T00:23:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2020-03-09T00:23:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: debug journal = 20 2020-03-09T00:23:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression algorithm = zlib 2020-03-09T00:23:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression mode = aggressive 2020-03-09T00:23:03.849 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify missing on start = True 2020-03-09T00:23:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 20 2020-03-09T00:23:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: osd shutdown pgref assert = True 2020-03-09T00:23:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2020-03-09T00:23:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2020-03-09T00:23:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2020-03-09T00:23:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue = debug_random 2020-03-09T00:23:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub min interval = 60 2020-03-09T00:23:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2020-03-09T00:23:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2020-03-09T00:23:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue cut off = debug_random 2020-03-09T00:23:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2020-03-09T00:23:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2020-03-09T00:23:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 25 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug reject backfill probability = 0.3 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify cached snaps = True 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: osd max markdown count = 1000 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: osd max backfills = 3 2020-03-09T00:23:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: debug filestore = 20 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2020-03-09T00:23:03.853 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2020-03-09T00:23:03.854 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2020-03-09T00:23:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2020-03-09T00:23:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2020-03-09T00:23:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2020-03-09T00:23:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2020-03-09T00:23:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.855 INFO:teuthology.orchestra.run.smithi050.stdout: 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2020-03-09T00:23:03.856 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2020-03-09T00:23:03.857 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2020-03-09T00:23:03.857 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2020-03-09T00:23:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: chdir = "" 2020-03-09T00:23:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-03-09T00:23:03.857 INFO:teuthology.orchestra.run.smithi139.stdout: auth supported = cephx 2020-03-09T00:23:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: filestore xattr use omap = true 2020-03-09T00:23:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: mon clock drift allowed = 1.000 2020-03-09T00:23:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.858 INFO:teuthology.orchestra.run.smithi139.stdout: osd crush chooseleaf type = 0 2020-03-09T00:23:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: auth debug = true 2020-03-09T00:23:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on old message = true 2020-03-09T00:23:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on bug = true 2020-03-09T00:23:03.859 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: mon max pg per osd = 10000 # >= luminous 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: mon pg warn max object skew = 0 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: # disable pg_autoscaler by default for new pools 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: osd_pool_default_pg_autoscale_mode = off 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default size = 2 2020-03-09T00:23:03.860 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow primary affinity = true 2020-03-09T00:23:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow pg remap = true 2020-03-09T00:23:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on legacy crush tunables = false 2020-03-09T00:23:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on crush straw calc version zero = false 2020-03-09T00:23:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on no sortbitwise = false 2020-03-09T00:23:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on osd down out interval zero = false 2020-03-09T00:23:03.861 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on too few osds = false 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_no_redundancy = false 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: osd default data pool replay window = 5 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.862 INFO:teuthology.orchestra.run.smithi139.stdout: mon allow pool delete = true 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: mon cluster log file level = debug 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: debug asserts on shutdown = true 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: mon host = "172.21.15.148,172.21.15.139,172.21.15.50" 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: ms tcp read timeout = 5 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: ms inject socket failures = 2500 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: mon client directed command retry = 5 2020-03-09T00:23:03.863 INFO:teuthology.orchestra.run.smithi139.stdout: osd_async_recovery_min_cost = 1 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = 036b76b6-511e-4ec6-9669-0a47550bd261 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout:[osd] 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: osd journal size = 100 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub load threshold = 5.0 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub max interval = 120 2020-03-09T00:23:03.864 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: osd recover clone overlap = true 2020-03-09T00:23:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: osd recovery max chunk = 1048576 2020-03-09T00:23:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug shutdown = true 2020-03-09T00:23:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug op order = true 2020-03-09T00:23:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify stray on activate = true 2020-03-09T00:23:03.865 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: osd open classes on start = true 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug pg log writeout = true 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: osd deep scrub update digest min age = 30 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: osd map max advance = 10 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.866 INFO:teuthology.orchestra.run.smithi139.stdout: journal zero on create = true 2020-03-09T00:23:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: filestore ondisk finisher threads = 3 2020-03-09T00:23:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: filestore apply finisher threads = 3 2020-03-09T00:23:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: bdev debug aio = true 2020-03-09T00:23:03.867 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug misdirected ops = true 2020-03-09T00:23:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: debug journal = 20 2020-03-09T00:23:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore compression algorithm = zlib 2020-03-09T00:23:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore compression mode = aggressive 2020-03-09T00:23:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify missing on start = True 2020-03-09T00:23:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 20 2020-03-09T00:23:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: osd shutdown pgref assert = True 2020-03-09T00:23:03.868 INFO:teuthology.orchestra.run.smithi139.stdout: debug monc = 20 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd nearfull ratio = 0.8 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: osd objectstore = bluestore 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: osd op queue = debug_random 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub min interval = 60 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: osd failsafe full ratio = 0.95 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd full ratio = 0.9 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: osd op queue cut off = debug_random 2020-03-09T00:23:03.869 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd backfillfull_ratio = 0.85 2020-03-09T00:23:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore fsck on mount = True 2020-03-09T00:23:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: debug osd = 25 2020-03-09T00:23:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: debug bluestore = 20 2020-03-09T00:23:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: debug bluefs = 20 2020-03-09T00:23:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug reject backfill probability = 0.3 2020-03-09T00:23:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: debug rocksdb = 10 2020-03-09T00:23:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify cached snaps = True 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: osd max markdown count = 1000 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: osd max backfills = 3 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: bluestore block size = 96636764160 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: debug filestore = 20 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr] 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2020-03-09T00:23:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: debug mgr = 20 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: mgr/telemetry/nag = false 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout:[mon] 2020-03-09T00:23:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2020-03-09T00:23:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2020-03-09T00:23:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: debug paxos = 20 2020-03-09T00:23:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2020-03-09T00:23:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: mon data avail warn = 5 2020-03-09T00:23:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: mon mgr mkfs grace = 240 2020-03-09T00:23:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd reporter subtree level = osd 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd prime pg temp = true 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout:[client] 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: rgw cache enabled = true 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable ops log = true 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable usage log = true 2020-03-09T00:23:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-03-09T00:23:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-03-09T00:23:03.875 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.a] 2020-03-09T00:23:03.875 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.c] 2020-03-09T00:23:03.875 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.b] 2020-03-09T00:23:03.877 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2020-03-09T00:23:03.877 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2020-03-09T00:23:03.877 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-03-09T00:23:03.877 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2020-03-09T00:23:03.877 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.878 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2020-03-09T00:23:03.878 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.878 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2020-03-09T00:23:03.878 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.878 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2020-03-09T00:23:03.878 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2020-03-09T00:23:03.878 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.879 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2020-03-09T00:23:03.879 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2020-03-09T00:23:03.881 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.882 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2020-03-09T00:23:03.882 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2020-03-09T00:23:03.882 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.882 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2020-03-09T00:23:03.882 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2020-03-09T00:23:03.882 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2020-03-09T00:23:03.883 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2020-03-09T00:23:03.884 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2020-03-09T00:23:03.884 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-03-09T00:23:03.884 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2020-03-09T00:23:03.884 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.884 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-03-09T00:23:03.884 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.884 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2020-03-09T00:23:03.885 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.885 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2020-03-09T00:23:03.885 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.885 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2020-03-09T00:23:03.885 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2020-03-09T00:23:03.885 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = "172.21.15.148,172.21.15.139,172.21.15.50" 2020-03-09T00:23:03.885 INFO:teuthology.orchestra.run.smithi148.stdout: ms tcp read timeout = 5 2020-03-09T00:23:03.886 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject socket failures = 2500 2020-03-09T00:23:03.886 INFO:teuthology.orchestra.run.smithi148.stdout: mon client directed command retry = 5 2020-03-09T00:23:03.886 INFO:teuthology.orchestra.run.smithi148.stdout: osd_async_recovery_min_cost = 1 2020-03-09T00:23:03.886 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = 036b76b6-511e-4ec6-9669-0a47550bd261 2020-03-09T00:23:03.886 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.886 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2020-03-09T00:23:03.887 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2020-03-09T00:23:03.887 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.887 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2020-03-09T00:23:03.887 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 120 2020-03-09T00:23:03.888 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.888 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2020-03-09T00:23:03.888 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2020-03-09T00:23:03.888 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.888 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2020-03-09T00:23:03.888 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2020-03-09T00:23:03.888 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2020-03-09T00:23:03.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2020-03-09T00:23:03.890 INFO:teuthology.orchestra.run.smithi148.stdout: debug journal = 20 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore compression algorithm = zlib 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore compression mode = aggressive 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify missing on start = True 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 20 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: osd shutdown pgref assert = True 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: debug monc = 20 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd nearfull ratio = 0.8 2020-03-09T00:23:03.891 INFO:teuthology.orchestra.run.smithi148.stdout: osd objectstore = bluestore 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue = debug_random 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub min interval = 60 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: osd failsafe full ratio = 0.95 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd full ratio = 0.9 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue cut off = debug_random 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd backfillfull_ratio = 0.85 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore fsck on mount = True 2020-03-09T00:23:03.892 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 25 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluestore = 20 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluefs = 20 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug reject backfill probability = 0.3 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: debug rocksdb = 10 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify cached snaps = True 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: osd max markdown count = 1000 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: osd max backfills = 3 2020-03-09T00:23:03.893 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore block size = 96636764160 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout: debug filestore = 20 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2020-03-09T00:23:03.894 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.895 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.895 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2020-03-09T00:23:03.895 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.895 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2020-03-09T00:23:03.895 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2020-03-09T00:23:03.895 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2020-03-09T00:23:03.895 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2020-03-09T00:23:03.896 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2020-03-09T00:23:03.896 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2020-03-09T00:23:03.896 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2020-03-09T00:23:03.896 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2020-03-09T00:23:03.896 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2020-03-09T00:23:03.896 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2020-03-09T00:23:03.896 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2020-03-09T00:23:03.897 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:23:03.897 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2020-03-09T00:23:03.897 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2020-03-09T00:23:03.897 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2020-03-09T00:23:03.897 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2020-03-09T00:23:03.897 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-03-09T00:23:03.897 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-03-09T00:23:03.898 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2020-03-09T00:23:03.898 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.c] 2020-03-09T00:23:03.898 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.b] 2020-03-09T00:23:03.900 INFO:tasks.ceph:Creating admin key on mon.a... 2020-03-09T00:23:03.900 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-03-09T00:23:03.951 INFO:tasks.ceph:Copying monmap to all nodes... 2020-03-09T00:23:03.982 DEBUG:teuthology.orchestra.remote:smithi148:/etc/ceph/ceph.keyring is 206B 2020-03-09T00:23:04.092 DEBUG:teuthology.orchestra.remote:smithi148:/home/ubuntu/cephtest/ceph.monmap is 379B 2020-03-09T00:23:04.142 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:23:04.143 INFO:teuthology.orchestra.run.smithi148:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-03-09T00:23:04.203 INFO:teuthology.orchestra.run.smithi148:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:04.288 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2020-03-09T00:23:04.288 INFO:teuthology.orchestra.run.smithi104:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-03-09T00:23:04.347 INFO:teuthology.orchestra.run.smithi104:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:04.433 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:23:04.433 INFO:teuthology.orchestra.run.smithi050:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-03-09T00:23:04.498 INFO:teuthology.orchestra.run.smithi050:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:04.583 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:23:04.583 INFO:teuthology.orchestra.run.smithi139:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-03-09T00:23:04.645 INFO:teuthology.orchestra.run.smithi139:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:04.737 INFO:tasks.ceph:Setting up mon nodes... 2020-03-09T00:23:04.738 INFO:tasks.ceph:Setting up mgr nodes... 2020-03-09T00:23:04.738 INFO:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2020-03-09T00:23:04.791 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2020-03-09T00:23:04.794 INFO:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-03-09T00:23:04.858 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-03-09T00:23:04.860 INFO:tasks.ceph:Setting up mds nodes... 2020-03-09T00:23:04.860 INFO:tasks.ceph_client:Setting up client nodes... 2020-03-09T00:23:04.861 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2020-03-09T00:23:04.900 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.client.0.keyring 2020-03-09T00:23:04.916 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-03-09T00:23:04.918 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_4', 'osd.12': '/dev/vg_nvme/lv_3', 'osd.4': '/dev/vg_nvme/lv_1', 'osd.8': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_4', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.13': '/dev/vg_nvme/lv_3', 'osd.5': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'osd.10': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_4', 'osd.14': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.11': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_4', 'osd.15': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_1'}} 2020-03-09T00:23:04.918 INFO:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-03-09T00:23:04.950 INFO:tasks.ceph:{'osd.10': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_4', 'osd.14': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:04.950 INFO:tasks.ceph:{} 2020-03-09T00:23:04.950 INFO:tasks.ceph:osd.2 2020-03-09T00:23:04.951 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:23:04.951 INFO:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-03-09T00:23:05.490 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:05.490 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 2020-03-09T00:23:05.491 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:05.491 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:05.491 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:05.491 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:05.492 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:05.492 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2020-03-09T00:23:05.493 INFO:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-2 2020-03-09T00:23:05.601 INFO:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-03-09T00:23:05.636 INFO:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2020-03-09T00:23:05.710 INFO:tasks.ceph:{'osd.10': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_4', 'osd.14': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:05.710 INFO:tasks.ceph:{} 2020-03-09T00:23:05.710 INFO:tasks.ceph:osd.6 2020-03-09T00:23:05.710 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:23:05.711 INFO:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-03-09T00:23:06.234 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:06.235 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:06.235 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:06.235 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 2020-03-09T00:23:06.235 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:06.235 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:06.235 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:06.236 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:06.236 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:06.236 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:06.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2020-03-09T00:23:06.237 INFO:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-6 2020-03-09T00:23:06.282 INFO:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2020-03-09T00:23:06.353 INFO:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2020-03-09T00:23:06.421 INFO:tasks.ceph:{'osd.10': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_4', 'osd.14': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:06.421 INFO:tasks.ceph:{} 2020-03-09T00:23:06.421 INFO:tasks.ceph:osd.10 2020-03-09T00:23:06.421 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:23:06.422 INFO:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-03-09T00:23:06.955 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:06.957 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:06.957 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:06.958 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 2020-03-09T00:23:06.958 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:06.958 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:06.958 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:06.958 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:06.958 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:06.958 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:06.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2020-03-09T00:23:06.960 INFO:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-10 2020-03-09T00:23:07.031 INFO:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2020-03-09T00:23:07.062 INFO:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2020-03-09T00:23:07.129 INFO:tasks.ceph:{'osd.10': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_4', 'osd.14': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:07.130 INFO:tasks.ceph:{} 2020-03-09T00:23:07.130 INFO:tasks.ceph:osd.14 2020-03-09T00:23:07.130 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:23:07.130 INFO:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-03-09T00:23:07.681 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:07.681 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:07.681 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:07.681 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 2020-03-09T00:23:07.681 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:07.682 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:07.682 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:07.682 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:07.682 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:07.682 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:07.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2020-03-09T00:23:07.683 INFO:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-14 2020-03-09T00:23:07.796 INFO:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2020-03-09T00:23:07.826 INFO:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:07.955 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:07.953+0000 7f7229aefec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-03-09T00:23:07.955 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:07.953+0000 7f7229aefec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-03-09T00:23:07.955 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:07.953+0000 7f7229aefec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-03-09T00:23:07.955 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:07.953+0000 7f7229aefec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-03-09T00:23:07.956 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:07.953+0000 7f7229aefec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-03-09T00:23:07.960 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:07.958+0000 7f7229aefec0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2020-03-09T00:23:10.300 INFO:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-03-09T00:23:10.335 INFO:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:10.424 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:10.422+0000 7fe260a16ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2020-03-09T00:23:10.424 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:10.422+0000 7fe260a16ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2020-03-09T00:23:10.424 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:10.422+0000 7fe260a16ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-03-09T00:23:10.424 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:10.422+0000 7fe260a16ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-03-09T00:23:10.425 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:10.422+0000 7fe260a16ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-03-09T00:23:10.425 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:10.423+0000 7fe260a16ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2020-03-09T00:23:12.761 INFO:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2020-03-09T00:23:12.794 INFO:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:12.882 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:12.879+0000 7fe0b0c60ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2020-03-09T00:23:12.882 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:12.880+0000 7fe0b0c60ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2020-03-09T00:23:12.882 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:12.880+0000 7fe0b0c60ec0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2020-03-09T00:23:12.882 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:12.880+0000 7fe0b0c60ec0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2020-03-09T00:23:12.882 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:12.880+0000 7fe0b0c60ec0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2020-03-09T00:23:12.883 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:12.880+0000 7fe0b0c60ec0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2020-03-09T00:23:15.216 INFO:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2020-03-09T00:23:15.247 INFO:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:15.335 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:15.333+0000 7fc3990d2ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2020-03-09T00:23:15.335 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:15.333+0000 7fc3990d2ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2020-03-09T00:23:15.335 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:15.333+0000 7fc3990d2ec0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2020-03-09T00:23:15.335 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:15.333+0000 7fc3990d2ec0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2020-03-09T00:23:15.335 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:15.333+0000 7fc3990d2ec0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2020-03-09T00:23:15.336 INFO:teuthology.orchestra.run.smithi139.stderr:2020-03-09T00:23:15.334+0000 7fc3990d2ec0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2020-03-09T00:23:17.668 INFO:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2020-03-09T00:23:17.700 INFO:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-03-09T00:23:17.733 INFO:tasks.ceph:{'osd.0': '/dev/vg_nvme/lv_4', 'osd.12': '/dev/vg_nvme/lv_3', 'osd.4': '/dev/vg_nvme/lv_1', 'osd.8': '/dev/vg_nvme/lv_2'} 2020-03-09T00:23:17.733 INFO:tasks.ceph:{} 2020-03-09T00:23:17.733 INFO:tasks.ceph:osd.0 2020-03-09T00:23:17.733 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:23:17.733 INFO:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-03-09T00:23:18.281 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2020-03-09T00:23:18.282 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:18.282 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:18.282 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:18.283 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:18.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2020-03-09T00:23:18.284 INFO:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2020-03-09T00:23:18.419 INFO:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-03-09T00:23:18.453 INFO:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2020-03-09T00:23:18.522 INFO:tasks.ceph:{'osd.0': '/dev/vg_nvme/lv_4', 'osd.12': '/dev/vg_nvme/lv_3', 'osd.4': '/dev/vg_nvme/lv_1', 'osd.8': '/dev/vg_nvme/lv_2'} 2020-03-09T00:23:18.522 INFO:tasks.ceph:{} 2020-03-09T00:23:18.522 INFO:tasks.ceph:osd.4 2020-03-09T00:23:18.522 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:23:18.523 INFO:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-03-09T00:23:19.033 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:19.033 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:19.033 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:19.033 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2020-03-09T00:23:19.033 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:19.034 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:19.034 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:19.034 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:19.034 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:19.034 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:19.035 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2020-03-09T00:23:19.035 INFO:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2020-03-09T00:23:19.110 INFO:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2020-03-09T00:23:19.145 INFO:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2020-03-09T00:23:19.214 INFO:tasks.ceph:{'osd.0': '/dev/vg_nvme/lv_4', 'osd.12': '/dev/vg_nvme/lv_3', 'osd.4': '/dev/vg_nvme/lv_1', 'osd.8': '/dev/vg_nvme/lv_2'} 2020-03-09T00:23:19.214 INFO:tasks.ceph:{} 2020-03-09T00:23:19.215 INFO:tasks.ceph:osd.8 2020-03-09T00:23:19.215 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:23:19.215 INFO:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-03-09T00:23:19.758 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:19.758 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:19.758 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:19.758 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2020-03-09T00:23:19.759 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:19.759 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:19.759 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:19.759 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:19.759 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:19.759 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:19.760 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2020-03-09T00:23:19.760 INFO:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-8 2020-03-09T00:23:19.859 INFO:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2020-03-09T00:23:19.890 INFO:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2020-03-09T00:23:19.958 INFO:tasks.ceph:{'osd.0': '/dev/vg_nvme/lv_4', 'osd.12': '/dev/vg_nvme/lv_3', 'osd.4': '/dev/vg_nvme/lv_1', 'osd.8': '/dev/vg_nvme/lv_2'} 2020-03-09T00:23:19.958 INFO:tasks.ceph:{} 2020-03-09T00:23:19.958 INFO:tasks.ceph:osd.12 2020-03-09T00:23:19.958 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com 2020-03-09T00:23:19.959 INFO:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-03-09T00:23:20.498 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:20.498 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:20.498 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:20.498 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 2020-03-09T00:23:20.498 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:20.499 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:20.499 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:20.499 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:20.499 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:20.499 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:20.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2020-03-09T00:23:20.500 INFO:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-12 2020-03-09T00:23:20.622 INFO:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2020-03-09T00:23:20.664 INFO:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:20.762+0000 7f70eb349ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-03-09T00:23:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:20.764+0000 7f70eb349ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-03-09T00:23:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:20.764+0000 7f70eb349ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-03-09T00:23:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:20.764+0000 7f70eb349ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-03-09T00:23:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:20.764+0000 7f70eb349ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-03-09T00:23:20.771 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:20.769+0000 7f70eb349ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2020-03-09T00:23:23.107 INFO:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-03-09T00:23:23.139 INFO:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:23.228 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:23.226+0000 7fe0aab0aec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2020-03-09T00:23:23.228 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:23.226+0000 7fe0aab0aec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2020-03-09T00:23:23.228 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:23.226+0000 7fe0aab0aec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-03-09T00:23:23.228 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:23.226+0000 7fe0aab0aec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-03-09T00:23:23.228 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:23.226+0000 7fe0aab0aec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-03-09T00:23:23.229 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:23.227+0000 7fe0aab0aec0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2020-03-09T00:23:25.563 INFO:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2020-03-09T00:23:25.594 INFO:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:25.681 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:25.679+0000 7fef734b5ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2020-03-09T00:23:25.682 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:25.680+0000 7fef734b5ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2020-03-09T00:23:25.682 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:25.680+0000 7fef734b5ec0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2020-03-09T00:23:25.682 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:25.680+0000 7fef734b5ec0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2020-03-09T00:23:25.682 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:25.680+0000 7fef734b5ec0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2020-03-09T00:23:25.682 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:25.680+0000 7fef734b5ec0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2020-03-09T00:23:28.013 INFO:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2020-03-09T00:23:28.046 INFO:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:28.134 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:28.132+0000 7f5f32b0fec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2020-03-09T00:23:28.134 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:28.132+0000 7f5f32b0fec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2020-03-09T00:23:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:28.132+0000 7f5f32b0fec0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2020-03-09T00:23:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:28.133+0000 7f5f32b0fec0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2020-03-09T00:23:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:28.133+0000 7f5f32b0fec0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2020-03-09T00:23:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:23:28.133+0000 7f5f32b0fec0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2020-03-09T00:23:30.472 INFO:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2020-03-09T00:23:30.503 INFO:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-03-09T00:23:30.541 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_4', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.13': '/dev/vg_nvme/lv_3', 'osd.5': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:30.542 INFO:tasks.ceph:{} 2020-03-09T00:23:30.542 INFO:tasks.ceph:osd.1 2020-03-09T00:23:30.542 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:23:30.542 INFO:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-03-09T00:23:30.641 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2020-03-09T00:23:30.641 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:30.642 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:30.642 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:30.642 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:30.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2020-03-09T00:23:30.738 INFO:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-1 2020-03-09T00:23:30.786 INFO:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-03-09T00:23:30.821 INFO:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2020-03-09T00:23:30.892 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_4', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.13': '/dev/vg_nvme/lv_3', 'osd.5': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:30.893 INFO:tasks.ceph:{} 2020-03-09T00:23:30.893 INFO:tasks.ceph:osd.5 2020-03-09T00:23:30.893 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:23:30.893 INFO:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-03-09T00:23:30.990 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:30.990 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:30.990 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:30.990 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2020-03-09T00:23:30.990 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:30.991 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:30.991 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:30.991 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:30.991 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:30.991 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:30.992 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2020-03-09T00:23:30.992 INFO:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2020-03-09T00:23:31.035 INFO:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2020-03-09T00:23:31.111 INFO:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2020-03-09T00:23:31.182 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_4', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.13': '/dev/vg_nvme/lv_3', 'osd.5': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:31.183 INFO:tasks.ceph:{} 2020-03-09T00:23:31.183 INFO:tasks.ceph:osd.9 2020-03-09T00:23:31.183 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:23:31.183 INFO:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-03-09T00:23:31.277 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:31.277 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:31.277 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:31.277 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2020-03-09T00:23:31.278 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:31.278 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:31.278 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:31.278 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:31.278 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:31.278 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:31.279 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2020-03-09T00:23:31.279 INFO:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-9 2020-03-09T00:23:31.323 INFO:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2020-03-09T00:23:31.398 INFO:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2020-03-09T00:23:31.469 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_4', 'osd.9': '/dev/vg_nvme/lv_2', 'osd.13': '/dev/vg_nvme/lv_3', 'osd.5': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:31.469 INFO:tasks.ceph:{} 2020-03-09T00:23:31.469 INFO:tasks.ceph:osd.13 2020-03-09T00:23:31.470 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2020-03-09T00:23:31.470 INFO:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-03-09T00:23:31.564 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:31.564 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:31.564 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:31.564 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2020-03-09T00:23:31.564 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:31.565 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:31.565 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:31.565 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:31.565 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:31.565 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:31.566 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2020-03-09T00:23:31.566 INFO:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-13 2020-03-09T00:23:31.609 INFO:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2020-03-09T00:23:31.684 INFO:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:31.770+0000 7f793be7aec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-03-09T00:23:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:31.770+0000 7f793be7aec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-03-09T00:23:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:31.770+0000 7f793be7aec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-03-09T00:23:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:31.770+0000 7f793be7aec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-03-09T00:23:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:31.770+0000 7f793be7aec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-03-09T00:23:31.772 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:31.770+0000 7f793be7aec0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2020-03-09T00:23:34.108 INFO:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-03-09T00:23:34.146 INFO:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:34.239 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:34.237+0000 7f771742eec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2020-03-09T00:23:34.239 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:34.237+0000 7f771742eec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2020-03-09T00:23:34.239 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:34.237+0000 7f771742eec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-03-09T00:23:34.239 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:34.237+0000 7f771742eec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-03-09T00:23:34.240 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:34.237+0000 7f771742eec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-03-09T00:23:34.240 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:34.237+0000 7f771742eec0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2020-03-09T00:23:36.576 INFO:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2020-03-09T00:23:36.612 INFO:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:36.704+0000 7f8bd9631ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2020-03-09T00:23:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:36.705+0000 7f8bd9631ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2020-03-09T00:23:36.707 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:36.705+0000 7f8bd9631ec0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2020-03-09T00:23:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:36.705+0000 7f8bd9631ec0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2020-03-09T00:23:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:36.705+0000 7f8bd9631ec0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2020-03-09T00:23:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:36.705+0000 7f8bd9631ec0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2020-03-09T00:23:39.046 INFO:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2020-03-09T00:23:39.082 INFO:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:39.173+0000 7fd427b41ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2020-03-09T00:23:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:39.173+0000 7fd427b41ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2020-03-09T00:23:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:39.173+0000 7fd427b41ec0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2020-03-09T00:23:39.175 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:39.173+0000 7fd427b41ec0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2020-03-09T00:23:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:39.173+0000 7fd427b41ec0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2020-03-09T00:23:39.176 INFO:teuthology.orchestra.run.smithi050.stderr:2020-03-09T00:23:39.173+0000 7fd427b41ec0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2020-03-09T00:23:41.511 INFO:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2020-03-09T00:23:41.547 INFO:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2020-03-09T00:23:41.582 INFO:tasks.ceph:{'osd.11': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_4', 'osd.15': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:41.583 INFO:tasks.ceph:{} 2020-03-09T00:23:41.583 INFO:tasks.ceph:osd.3 2020-03-09T00:23:41.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com 2020-03-09T00:23:41.583 INFO:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-03-09T00:23:42.126 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:42.126 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:42.126 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:42.126 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 2020-03-09T00:23:42.126 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:42.127 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:42.127 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:42.127 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:42.127 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:42.127 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:42.128 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2020-03-09T00:23:42.128 INFO:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2020-03-09T00:23:42.243 INFO:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2020-03-09T00:23:42.278 INFO:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2020-03-09T00:23:42.348 INFO:tasks.ceph:{'osd.11': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_4', 'osd.15': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:42.348 INFO:tasks.ceph:{} 2020-03-09T00:23:42.348 INFO:tasks.ceph:osd.7 2020-03-09T00:23:42.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2020-03-09T00:23:42.349 INFO:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-03-09T00:23:42.878 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:42.879 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:42.879 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:42.879 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 2020-03-09T00:23:42.879 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:42.879 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:42.879 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:42.880 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:42.880 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:42.880 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:42.881 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2020-03-09T00:23:42.881 INFO:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7 2020-03-09T00:23:42.926 INFO:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2020-03-09T00:23:42.958 INFO:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2020-03-09T00:23:43.025 INFO:tasks.ceph:{'osd.11': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_4', 'osd.15': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:43.025 INFO:tasks.ceph:{} 2020-03-09T00:23:43.025 INFO:tasks.ceph:osd.11 2020-03-09T00:23:43.025 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2020-03-09T00:23:43.026 INFO:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-03-09T00:23:43.578 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:43.578 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:43.578 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:43.578 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 2020-03-09T00:23:43.579 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:43.579 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:43.579 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:43.579 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:43.579 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:43.579 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:43.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2020-03-09T00:23:43.580 INFO:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-11 2020-03-09T00:23:43.652 INFO:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2020-03-09T00:23:43.682 INFO:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2020-03-09T00:23:43.749 INFO:tasks.ceph:{'osd.11': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_4', 'osd.15': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_1'} 2020-03-09T00:23:43.749 INFO:tasks.ceph:{} 2020-03-09T00:23:43.749 INFO:tasks.ceph:osd.15 2020-03-09T00:23:43.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2020-03-09T00:23:43.750 INFO:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-03-09T00:23:44.317 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-03-09T00:23:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-09T00:23:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-03-09T00:23:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 2020-03-09T00:23:44.318 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-09T00:23:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2020-03-09T00:23:44.319 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-03-09T00:23:44.319 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-09T00:23:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-09T00:23:44.319 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-09T00:23:44.320 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2020-03-09T00:23:44.320 INFO:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-15 2020-03-09T00:23:44.445 INFO:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2020-03-09T00:23:44.475 INFO:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:44.558 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:44.556+0000 7fd861844ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2020-03-09T00:23:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:44.559+0000 7fd861844ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2020-03-09T00:23:44.561 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:44.560+0000 7fd861844ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-03-09T00:23:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:44.560+0000 7fd861844ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-03-09T00:23:44.562 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:44.560+0000 7fd861844ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-03-09T00:23:44.571 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:44.570+0000 7fd861844ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2020-03-09T00:23:46.912 INFO:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2020-03-09T00:23:46.985 INFO:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:47.071+0000 7fae33f73ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2020-03-09T00:23:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:47.071+0000 7fae33f73ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2020-03-09T00:23:47.073 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:47.071+0000 7fae33f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-03-09T00:23:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:47.071+0000 7fae33f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-03-09T00:23:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:47.071+0000 7fae33f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-03-09T00:23:47.074 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:47.072+0000 7fae33f73ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2020-03-09T00:23:49.410 INFO:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2020-03-09T00:23:49.441 INFO:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:49.529 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:49.527+0000 7f9036977ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2020-03-09T00:23:49.529 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:49.527+0000 7f9036977ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2020-03-09T00:23:49.529 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:49.527+0000 7f9036977ec0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2020-03-09T00:23:49.529 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:49.527+0000 7f9036977ec0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2020-03-09T00:23:49.529 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:49.527+0000 7f9036977ec0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2020-03-09T00:23:49.530 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:49.527+0000 7f9036977ec0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2020-03-09T00:23:51.868 INFO:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2020-03-09T00:23:51.899 INFO:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:23:51.988 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:51.985+0000 7feb54a7bec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2020-03-09T00:23:51.988 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:51.986+0000 7feb54a7bec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2020-03-09T00:23:51.988 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:51.986+0000 7feb54a7bec0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2020-03-09T00:23:51.988 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:51.986+0000 7feb54a7bec0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2020-03-09T00:23:51.988 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:51.986+0000 7feb54a7bec0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2020-03-09T00:23:51.989 INFO:teuthology.orchestra.run.smithi104.stderr:2020-03-09T00:23:51.986+0000 7feb54a7bec0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2020-03-09T00:23:54.328 INFO:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2020-03-09T00:23:54.360 INFO:tasks.ceph:Reading keys from all nodes... 2020-03-09T00:23:54.360 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:23:54.378 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.6QEZAvuW9K 2020-03-09T00:23:54.378 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.6QEZAvuW9K 2020-03-09T00:23:54.451 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.6QEZAvuW9K 2020-03-09T00:23:54.580 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.6QEZAvuW9K is 56B 2020-03-09T00:23:54.643 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.6QEZAvuW9K 2020-03-09T00:23:54.658 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:23:54.712 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.RgnkKtEGNv 2020-03-09T00:23:54.712 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.RgnkKtEGNv 2020-03-09T00:23:54.781 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.RgnkKtEGNv 2020-03-09T00:23:54.908 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.RgnkKtEGNv is 56B 2020-03-09T00:23:54.958 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.RgnkKtEGNv 2020-03-09T00:23:54.972 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:23:55.025 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.vYfuvlwgq6 2020-03-09T00:23:55.025 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.vYfuvlwgq6 2020-03-09T00:23:55.093 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.vYfuvlwgq6 2020-03-09T00:23:55.217 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.vYfuvlwgq6 is 56B 2020-03-09T00:23:55.264 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.vYfuvlwgq6 2020-03-09T00:23:55.279 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:23:55.333 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.TL2LaJQh1R 2020-03-09T00:23:55.333 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-8/keyring /tmp/tmp.TL2LaJQh1R 2020-03-09T00:23:55.400 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.TL2LaJQh1R 2020-03-09T00:23:55.524 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.TL2LaJQh1R is 56B 2020-03-09T00:23:55.572 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.TL2LaJQh1R 2020-03-09T00:23:55.585 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:23:55.637 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.Nw09tFbHsu 2020-03-09T00:23:55.638 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-12/keyring /tmp/tmp.Nw09tFbHsu 2020-03-09T00:23:55.704 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.Nw09tFbHsu 2020-03-09T00:23:55.828 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.Nw09tFbHsu is 57B 2020-03-09T00:23:55.877 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.Nw09tFbHsu 2020-03-09T00:23:55.890 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:23:55.907 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.oUkGwCQwZW 2020-03-09T00:23:55.907 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.oUkGwCQwZW 2020-03-09T00:23:55.977 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.oUkGwCQwZW 2020-03-09T00:23:56.105 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.oUkGwCQwZW is 56B 2020-03-09T00:23:56.153 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.oUkGwCQwZW 2020-03-09T00:23:56.169 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:23:56.222 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.fzAe3WTvNf 2020-03-09T00:23:56.223 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.fzAe3WTvNf 2020-03-09T00:23:56.290 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.fzAe3WTvNf 2020-03-09T00:23:56.414 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.fzAe3WTvNf is 56B 2020-03-09T00:23:56.462 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.fzAe3WTvNf 2020-03-09T00:23:56.476 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:23:56.530 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.gaEn0sYWxP 2020-03-09T00:23:56.530 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.gaEn0sYWxP 2020-03-09T00:23:56.598 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.gaEn0sYWxP 2020-03-09T00:23:56.722 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.gaEn0sYWxP is 56B 2020-03-09T00:23:56.770 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.gaEn0sYWxP 2020-03-09T00:23:56.784 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:23:56.837 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.aaK73PvY5Q 2020-03-09T00:23:56.838 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-11/keyring /tmp/tmp.aaK73PvY5Q 2020-03-09T00:23:56.905 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.aaK73PvY5Q 2020-03-09T00:23:57.028 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.aaK73PvY5Q is 57B 2020-03-09T00:23:57.075 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.aaK73PvY5Q 2020-03-09T00:23:57.088 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:23:57.141 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.PsGzEwEdiM 2020-03-09T00:23:57.141 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-15/keyring /tmp/tmp.PsGzEwEdiM 2020-03-09T00:23:57.208 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.PsGzEwEdiM 2020-03-09T00:23:57.332 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.PsGzEwEdiM is 57B 2020-03-09T00:23:57.380 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.PsGzEwEdiM 2020-03-09T00:23:57.393 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:23:57.410 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.KY1nBcIEh0 2020-03-09T00:23:57.410 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.KY1nBcIEh0 2020-03-09T00:23:57.487 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.KY1nBcIEh0 2020-03-09T00:23:57.621 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.KY1nBcIEh0 is 56B 2020-03-09T00:23:57.669 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.KY1nBcIEh0 2020-03-09T00:23:57.684 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:23:57.738 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.cYTMqgKEoh 2020-03-09T00:23:57.739 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.cYTMqgKEoh 2020-03-09T00:23:57.812 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.cYTMqgKEoh 2020-03-09T00:23:57.944 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.cYTMqgKEoh is 56B 2020-03-09T00:23:57.992 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.cYTMqgKEoh 2020-03-09T00:23:58.008 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:23:58.062 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.XqoRD4qz3F 2020-03-09T00:23:58.063 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-9/keyring /tmp/tmp.XqoRD4qz3F 2020-03-09T00:23:58.135 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.XqoRD4qz3F 2020-03-09T00:23:58.265 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.XqoRD4qz3F is 56B 2020-03-09T00:23:58.315 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.XqoRD4qz3F 2020-03-09T00:23:58.330 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:23:58.383 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.phFY1cxvt2 2020-03-09T00:23:58.385 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-13/keyring /tmp/tmp.phFY1cxvt2 2020-03-09T00:23:58.454 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.phFY1cxvt2 2020-03-09T00:23:58.588 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.phFY1cxvt2 is 57B 2020-03-09T00:23:58.636 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.phFY1cxvt2 2020-03-09T00:23:58.650 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:23:58.667 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.xXkEHsBQFt 2020-03-09T00:23:58.668 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.xXkEHsBQFt 2020-03-09T00:23:58.740 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.xXkEHsBQFt 2020-03-09T00:23:58.868 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.xXkEHsBQFt is 56B 2020-03-09T00:23:58.916 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.xXkEHsBQFt 2020-03-09T00:23:58.930 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:23:58.983 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.IPtVVSJ5iS 2020-03-09T00:23:58.984 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.IPtVVSJ5iS 2020-03-09T00:23:59.051 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.IPtVVSJ5iS 2020-03-09T00:23:59.175 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.IPtVVSJ5iS is 56B 2020-03-09T00:23:59.224 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.IPtVVSJ5iS 2020-03-09T00:23:59.238 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:23:59.290 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.IWJMbgsGLo 2020-03-09T00:23:59.290 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-10/keyring /tmp/tmp.IWJMbgsGLo 2020-03-09T00:23:59.358 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.IWJMbgsGLo 2020-03-09T00:23:59.479 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.IWJMbgsGLo is 57B 2020-03-09T00:23:59.526 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.IWJMbgsGLo 2020-03-09T00:23:59.539 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:23:59.591 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.6iJWLHVuLb 2020-03-09T00:23:59.592 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-14/keyring /tmp/tmp.6iJWLHVuLb 2020-03-09T00:23:59.658 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.6iJWLHVuLb 2020-03-09T00:23:59.782 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.6iJWLHVuLb is 57B 2020-03-09T00:23:59.831 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.6iJWLHVuLb 2020-03-09T00:23:59.865 DEBUG:teuthology.orchestra.remote:smithi104:/etc/ceph/ceph.client.0.keyring is 59B 2020-03-09T00:23:59.913 INFO:tasks.ceph:Adding keys to all mons... 2020-03-09T00:23:59.913 INFO:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2020-03-09T00:23:59.916 INFO:teuthology.orchestra.run.smithi139:> sudo tee -a /etc/ceph/ceph.keyring 2020-03-09T00:23:59.918 INFO:teuthology.orchestra.run.smithi148:> sudo tee -a /etc/ceph/ceph.keyring 2020-03-09T00:23:59.941 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.y] 2020-03-09T00:23:59.941 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBojGVeeFUoLxAADdLn3diwvFJ8mLEDx+hN1Q== 2020-03-09T00:23:59.941 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2020-03-09T00:23:59.941 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQB4jGVep4uGLRAAg768mYHkWY4G7JCIQ8XFAg== 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.4] 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQB7jGVeHrOIDRAA3eAbpPGPrYkCBy4ta2XFLQ== 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.8] 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQB9jGVeF1ePKBAA/J+i4oFs4YDNo38tG6s+6g== 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.12] 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCAjGVeD8vxBxAAg6k7RAQDhyx/LWSawyMI7Q== 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.x] 2020-03-09T00:23:59.942 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBojGVe5X4iMxAAHUSqH4WUfbVhpujw6IjQRQ== 2020-03-09T00:23:59.943 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.3] 2020-03-09T00:23:59.943 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCQjGVel8Y9IRAAD/9MqbQCTkNrAhEBRP8/4A== 2020-03-09T00:23:59.943 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.7] 2020-03-09T00:23:59.943 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCTjGVejOFPBBAAyMsr/iEgr1SKzX6uru659A== 2020-03-09T00:23:59.943 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.11] 2020-03-09T00:23:59.943 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCVjGVervl2HxAA9y1JpOfXhmF5Mqc+M7spDA== 2020-03-09T00:23:59.943 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.15] 2020-03-09T00:23:59.944 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCXjGVe/hLROhAA8us1RyRbUT9qxt7apsB5sw== 2020-03-09T00:23:59.944 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2020-03-09T00:23:59.944 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCDjGVeo63vLRAAU+pvy62ugQGoHCNh2XJCZg== 2020-03-09T00:23:59.944 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.5] 2020-03-09T00:23:59.944 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCGjGVe2YktDhAAgICaBal8oEn7ujLThWZK6g== 2020-03-09T00:23:59.944 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.9] 2020-03-09T00:23:59.944 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCIjGVetrUNKhAAavpq5GwoIQl29yHKsKOGWQ== 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.13] 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQCLjGVecpJeChAAL23fSLpSezebYzwdVskLWA== 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBrjGVeAAHfOBAAi59l76jm4MbucRgEfNIwrQ== 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.6] 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBujGVeOcI2GRAA0QSkANwpqe/mLe/HAL3hRw== 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.10] 2020-03-09T00:23:59.945 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBwjGVeXzV+NBAAj1IH8isD/XYuoCJn9SWG8Q== 2020-03-09T00:23:59.946 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.14] 2020-03-09T00:23:59.946 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBzjGVehw7lExAAiLrT5lqcc32BOK2jPYh9Lg== 2020-03-09T00:23:59.946 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2020-03-09T00:23:59.946 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBojGVeU12fNRAAHj8GixulXVw+P+46EKV6xA== 2020-03-09T00:23:59.948 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.y] 2020-03-09T00:23:59.948 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBojGVeeFUoLxAADdLn3diwvFJ8mLEDx+hN1Q== 2020-03-09T00:23:59.948 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2020-03-09T00:23:59.948 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQB4jGVep4uGLRAAg768mYHkWY4G7JCIQ8XFAg== 2020-03-09T00:23:59.948 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2020-03-09T00:23:59.948 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQB7jGVeHrOIDRAA3eAbpPGPrYkCBy4ta2XFLQ== 2020-03-09T00:23:59.948 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.8] 2020-03-09T00:23:59.949 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQB9jGVeF1ePKBAA/J+i4oFs4YDNo38tG6s+6g== 2020-03-09T00:23:59.949 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.12] 2020-03-09T00:23:59.949 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCAjGVeD8vxBxAAg6k7RAQDhyx/LWSawyMI7Q== 2020-03-09T00:23:59.949 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2020-03-09T00:23:59.949 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBojGVe5X4iMxAAHUSqH4WUfbVhpujw6IjQRQ== 2020-03-09T00:23:59.949 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2020-03-09T00:23:59.949 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCQjGVel8Y9IRAAD/9MqbQCTkNrAhEBRP8/4A== 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.7] 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCTjGVejOFPBBAAyMsr/iEgr1SKzX6uru659A== 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.11] 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCVjGVervl2HxAA9y1JpOfXhmF5Mqc+M7spDA== 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.15] 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCXjGVe/hLROhAA8us1RyRbUT9qxt7apsB5sw== 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2020-03-09T00:23:59.950 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCDjGVeo63vLRAAU+pvy62ugQGoHCNh2XJCZg== 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCGjGVe2YktDhAAgICaBal8oEn7ujLThWZK6g== 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.9] 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCIjGVetrUNKhAAavpq5GwoIQl29yHKsKOGWQ== 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.13] 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCLjGVecpJeChAAL23fSLpSezebYzwdVskLWA== 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2020-03-09T00:23:59.951 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBrjGVeAAHfOBAAi59l76jm4MbucRgEfNIwrQ== 2020-03-09T00:23:59.952 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.6] 2020-03-09T00:23:59.952 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBujGVeOcI2GRAA0QSkANwpqe/mLe/HAL3hRw== 2020-03-09T00:23:59.952 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.10] 2020-03-09T00:23:59.952 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBwjGVeXzV+NBAAj1IH8isD/XYuoCJn9SWG8Q== 2020-03-09T00:23:59.952 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.14] 2020-03-09T00:23:59.953 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBzjGVehw7lExAAiLrT5lqcc32BOK2jPYh9Lg== 2020-03-09T00:23:59.953 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2020-03-09T00:23:59.953 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBojGVeU12fNRAAHj8GixulXVw+P+46EKV6xA== 2020-03-09T00:23:59.953 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.y] 2020-03-09T00:23:59.954 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBojGVeeFUoLxAADdLn3diwvFJ8mLEDx+hN1Q== 2020-03-09T00:23:59.954 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2020-03-09T00:23:59.954 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQB4jGVep4uGLRAAg768mYHkWY4G7JCIQ8XFAg== 2020-03-09T00:23:59.954 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.4] 2020-03-09T00:23:59.954 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQB7jGVeHrOIDRAA3eAbpPGPrYkCBy4ta2XFLQ== 2020-03-09T00:23:59.954 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.8] 2020-03-09T00:23:59.954 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQB9jGVeF1ePKBAA/J+i4oFs4YDNo38tG6s+6g== 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.12] 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCAjGVeD8vxBxAAg6k7RAQDhyx/LWSawyMI7Q== 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.x] 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBojGVe5X4iMxAAHUSqH4WUfbVhpujw6IjQRQ== 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.3] 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCQjGVel8Y9IRAAD/9MqbQCTkNrAhEBRP8/4A== 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.7] 2020-03-09T00:23:59.955 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCTjGVejOFPBBAAyMsr/iEgr1SKzX6uru659A== 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.11] 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCVjGVervl2HxAA9y1JpOfXhmF5Mqc+M7spDA== 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.15] 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCXjGVe/hLROhAA8us1RyRbUT9qxt7apsB5sw== 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCDjGVeo63vLRAAU+pvy62ugQGoHCNh2XJCZg== 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.5] 2020-03-09T00:23:59.956 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCGjGVe2YktDhAAgICaBal8oEn7ujLThWZK6g== 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.9] 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCIjGVetrUNKhAAavpq5GwoIQl29yHKsKOGWQ== 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.13] 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCLjGVecpJeChAAL23fSLpSezebYzwdVskLWA== 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBrjGVeAAHfOBAAi59l76jm4MbucRgEfNIwrQ== 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.6] 2020-03-09T00:23:59.957 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBujGVeOcI2GRAA0QSkANwpqe/mLe/HAL3hRw== 2020-03-09T00:23:59.958 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.10] 2020-03-09T00:23:59.958 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBwjGVeXzV+NBAAj1IH8isD/XYuoCJn9SWG8Q== 2020-03-09T00:23:59.958 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.14] 2020-03-09T00:23:59.958 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBzjGVehw7lExAAiLrT5lqcc32BOK2jPYh9Lg== 2020-03-09T00:23:59.958 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2020-03-09T00:23:59.958 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQBojGVeU12fNRAAHj8GixulXVw+P+46EKV6xA== 2020-03-09T00:23:59.960 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-09T00:24:00.002 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-09T00:24:00.004 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-09T00:24:00.058 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.060 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.063 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.113 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.116 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.120 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.170 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.172 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.206 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.254 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.256 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.260 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.307 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-09T00:24:00.309 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-09T00:24:00.312 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-09T00:24:00.359 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.362 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.364 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.412 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.414 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.416 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.464 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.466 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.468 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.516 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.518 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.520 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.568 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.570 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.573 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.621 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.623 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.625 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.673 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.675 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.678 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.726 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.728 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.730 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.778 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.780 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.783 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.830 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.833 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.835 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.883 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.886 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.888 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.936 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.939 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.942 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-09T00:24:00.991 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-03-09T00:24:00.994 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-03-09T00:24:00.996 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-03-09T00:24:01.047 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-03-09T00:24:01.048 INFO:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-03-09T00:24:01.076 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-03-09T00:24:01.168 INFO:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-03-09T00:24:01.195 INFO:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-03-09T00:24:01.227 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-03-09T00:24:01.324 INFO:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-03-09T00:24:01.354 INFO:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-03-09T00:24:01.383 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-03-09T00:24:01.475 INFO:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-03-09T00:24:01.502 INFO:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:24:01.504 INFO:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:24:01.545 INFO:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-03-09T00:24:01.559 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-03-09T00:24:01.559 INFO:tasks.ceph.mon.a:Restarting daemon 2020-03-09T00:24:01.560 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-03-09T00:24:01.602 INFO:tasks.ceph.mon.a:Started 2020-03-09T00:24:01.602 INFO:tasks.ceph.mon.b:Restarting daemon 2020-03-09T00:24:01.603 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-03-09T00:24:01.605 INFO:tasks.ceph.mon.b:Started 2020-03-09T00:24:01.605 INFO:tasks.ceph.mon.c:Restarting daemon 2020-03-09T00:24:01.605 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-03-09T00:24:01.607 INFO:tasks.ceph.mon.c:Started 2020-03-09T00:24:01.607 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-03-09T00:24:01.608 INFO:tasks.ceph.mgr.y:Restarting daemon 2020-03-09T00:24:01.608 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2020-03-09T00:24:01.610 INFO:tasks.ceph.mgr.y:Started 2020-03-09T00:24:01.610 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-03-09T00:24:01.610 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-03-09T00:24:01.613 INFO:tasks.ceph.mgr.x:Started 2020-03-09T00:24:01.613 INFO:tasks.ceph:Setting crush tunables to default 2020-03-09T00:24:01.613 INFO:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd crush tunables default 2020-03-09T00:24:06.853 INFO:teuthology.orchestra.run.smithi148.stderr:adjusted tunables profile to default 2020-03-09T00:24:06.863 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-03-09T00:24:06.863 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:24:06.880 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.2VHb8ISHR3 2020-03-09T00:24:06.881 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.2VHb8ISHR3 2020-03-09T00:24:06.951 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.2VHb8ISHR3 2020-03-09T00:24:07.079 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.2VHb8ISHR3 is 37B 2020-03-09T00:24:07.128 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.2VHb8ISHR3 2020-03-09T00:24:07.143 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:24:07.196 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.Se7BaeUy2m 2020-03-09T00:24:07.198 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.Se7BaeUy2m 2020-03-09T00:24:07.265 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.Se7BaeUy2m 2020-03-09T00:24:07.391 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.Se7BaeUy2m is 37B 2020-03-09T00:24:07.439 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.Se7BaeUy2m 2020-03-09T00:24:07.452 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:24:07.506 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.51Mrns8PHa 2020-03-09T00:24:07.506 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-10/fsid /tmp/tmp.51Mrns8PHa 2020-03-09T00:24:07.574 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.51Mrns8PHa 2020-03-09T00:24:07.877 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.51Mrns8PHa is 37B 2020-03-09T00:24:07.924 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.51Mrns8PHa 2020-03-09T00:24:07.938 INFO:teuthology.orchestra.run.smithi139:> mktemp 2020-03-09T00:24:07.991 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.ABQCLtKCYR 2020-03-09T00:24:07.991 INFO:teuthology.orchestra.run.smithi139:> sudo cp /var/lib/ceph/osd/ceph-14/fsid /tmp/tmp.ABQCLtKCYR 2020-03-09T00:24:08.058 INFO:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.ABQCLtKCYR 2020-03-09T00:24:08.182 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.ABQCLtKCYR is 37B 2020-03-09T00:24:08.230 INFO:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.ABQCLtKCYR 2020-03-09T00:24:08.243 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:24:08.257 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.I1y9rjFKHz 2020-03-09T00:24:08.258 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.I1y9rjFKHz 2020-03-09T00:24:08.325 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.I1y9rjFKHz 2020-03-09T00:24:08.449 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.I1y9rjFKHz is 37B 2020-03-09T00:24:08.467 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.I1y9rjFKHz 2020-03-09T00:24:08.481 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:24:08.534 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.uCqeLAw7Lk 2020-03-09T00:24:08.535 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.uCqeLAw7Lk 2020-03-09T00:24:08.602 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.uCqeLAw7Lk 2020-03-09T00:24:08.743 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.uCqeLAw7Lk is 37B 2020-03-09T00:24:08.761 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.uCqeLAw7Lk 2020-03-09T00:24:08.775 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:24:08.828 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.1rZ3yjeS3T 2020-03-09T00:24:08.828 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-8/fsid /tmp/tmp.1rZ3yjeS3T 2020-03-09T00:24:08.896 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.1rZ3yjeS3T 2020-03-09T00:24:09.021 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.1rZ3yjeS3T is 37B 2020-03-09T00:24:09.040 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.1rZ3yjeS3T 2020-03-09T00:24:09.054 INFO:teuthology.orchestra.run.smithi148:> mktemp 2020-03-09T00:24:09.107 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.Ep0JcIdwed 2020-03-09T00:24:09.107 INFO:teuthology.orchestra.run.smithi148:> sudo cp /var/lib/ceph/osd/ceph-12/fsid /tmp/tmp.Ep0JcIdwed 2020-03-09T00:24:09.175 INFO:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.Ep0JcIdwed 2020-03-09T00:24:09.299 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.Ep0JcIdwed is 37B 2020-03-09T00:24:09.346 INFO:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.Ep0JcIdwed 2020-03-09T00:24:09.359 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:24:09.377 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.y1jZO9By7x 2020-03-09T00:24:09.377 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.y1jZO9By7x 2020-03-09T00:24:09.453 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.y1jZO9By7x 2020-03-09T00:24:09.587 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.y1jZO9By7x is 37B 2020-03-09T00:24:09.636 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.y1jZO9By7x 2020-03-09T00:24:09.651 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:24:09.706 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.41MllaU3bL 2020-03-09T00:24:09.707 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.41MllaU3bL 2020-03-09T00:24:09.779 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.41MllaU3bL 2020-03-09T00:24:09.910 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.41MllaU3bL is 37B 2020-03-09T00:24:09.960 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.41MllaU3bL 2020-03-09T00:24:09.977 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:24:10.028 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.HKbZxv9ojW 2020-03-09T00:24:10.029 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-9/fsid /tmp/tmp.HKbZxv9ojW 2020-03-09T00:24:10.100 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.HKbZxv9ojW 2020-03-09T00:24:10.228 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.HKbZxv9ojW is 37B 2020-03-09T00:24:10.277 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.HKbZxv9ojW 2020-03-09T00:24:10.291 INFO:teuthology.orchestra.run.smithi050:> mktemp 2020-03-09T00:24:10.344 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.p7vgPOCCch 2020-03-09T00:24:10.345 INFO:teuthology.orchestra.run.smithi050:> sudo cp /var/lib/ceph/osd/ceph-13/fsid /tmp/tmp.p7vgPOCCch 2020-03-09T00:24:10.415 INFO:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.p7vgPOCCch 2020-03-09T00:24:10.542 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.p7vgPOCCch is 37B 2020-03-09T00:24:10.591 INFO:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.p7vgPOCCch 2020-03-09T00:24:10.604 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:24:10.621 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.JbKDWyoIoP 2020-03-09T00:24:10.622 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.JbKDWyoIoP 2020-03-09T00:24:10.693 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.JbKDWyoIoP 2020-03-09T00:24:10.822 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.JbKDWyoIoP is 37B 2020-03-09T00:24:10.870 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.JbKDWyoIoP 2020-03-09T00:24:10.885 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:24:10.938 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.jroxGs4ZCz 2020-03-09T00:24:10.939 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.jroxGs4ZCz 2020-03-09T00:24:11.006 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.jroxGs4ZCz 2020-03-09T00:24:11.130 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.jroxGs4ZCz is 37B 2020-03-09T00:24:11.178 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.jroxGs4ZCz 2020-03-09T00:24:11.191 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:24:11.244 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.0KzluC7yE4 2020-03-09T00:24:11.244 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-11/fsid /tmp/tmp.0KzluC7yE4 2020-03-09T00:24:11.310 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.0KzluC7yE4 2020-03-09T00:24:11.433 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.0KzluC7yE4 is 37B 2020-03-09T00:24:11.482 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.0KzluC7yE4 2020-03-09T00:24:11.495 INFO:teuthology.orchestra.run.smithi104:> mktemp 2020-03-09T00:24:11.548 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.mBAJw9SkFW 2020-03-09T00:24:11.548 INFO:teuthology.orchestra.run.smithi104:> sudo cp /var/lib/ceph/osd/ceph-15/fsid /tmp/tmp.mBAJw9SkFW 2020-03-09T00:24:11.616 INFO:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.mBAJw9SkFW 2020-03-09T00:24:11.741 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.mBAJw9SkFW is 37B 2020-03-09T00:24:11.789 INFO:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.mBAJw9SkFW 2020-03-09T00:24:11.802 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new d0be76d1-3ce6-4c23-9917-41b35482def3 0 2020-03-09T00:24:12.110 INFO:teuthology.orchestra.run.smithi104.stdout:0 2020-03-09T00:24:12.121 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new cee70020-657e-4b62-89cb-226f760442f5 1 2020-03-09T00:24:12.389 INFO:teuthology.orchestra.run.smithi104.stdout:1 2020-03-09T00:24:12.400 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 4103b4d7-7a38-48e4-8403-3549f5b3fda2 2 2020-03-09T00:24:12.672 INFO:teuthology.orchestra.run.smithi104.stdout:2 2020-03-09T00:24:12.684 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new ce80df27-1b2e-41e1-9ded-a012ad488591 3 2020-03-09T00:24:12.955 INFO:teuthology.orchestra.run.smithi104.stdout:3 2020-03-09T00:24:12.966 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 8ff5a871-6f10-4e55-ac60-33bdc11c0292 4 2020-03-09T00:24:13.242 INFO:teuthology.orchestra.run.smithi104.stdout:4 2020-03-09T00:24:13.253 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 5122cc94-7217-4d84-8b81-11383b48c416 5 2020-03-09T00:24:13.530 INFO:teuthology.orchestra.run.smithi104.stdout:5 2020-03-09T00:24:13.542 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 1bea4f99-5e2a-4fbc-bac3-8cb45d678742 6 2020-03-09T00:24:13.814 INFO:teuthology.orchestra.run.smithi104.stdout:6 2020-03-09T00:24:13.825 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new f6a853dd-1462-4f8c-83d6-701258d78519 7 2020-03-09T00:24:14.092 INFO:teuthology.orchestra.run.smithi104.stdout:7 2020-03-09T00:24:14.104 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new b9911260-c1d0-4563-892b-35a9e1c47a9b 8 2020-03-09T00:24:14.373 INFO:teuthology.orchestra.run.smithi104.stdout:8 2020-03-09T00:24:14.384 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new d7ebf4f6-a2b8-4076-83dd-504172481e32 9 2020-03-09T00:24:14.652 INFO:teuthology.orchestra.run.smithi104.stdout:9 2020-03-09T00:24:14.663 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new cb76df3e-6ea9-4ee6-8b03-84e3cd329c69 10 2020-03-09T00:24:14.932 INFO:teuthology.orchestra.run.smithi104.stdout:10 2020-03-09T00:24:14.944 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 8bda27ba-cf17-472c-9a21-02dcaefab68f 11 2020-03-09T00:24:15.219 INFO:teuthology.orchestra.run.smithi104.stdout:11 2020-03-09T00:24:15.231 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new d861da04-0be8-434b-9ed8-c1ec97fca7bd 12 2020-03-09T00:24:15.504 INFO:teuthology.orchestra.run.smithi104.stdout:12 2020-03-09T00:24:15.515 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 4ba3410e-df5b-4529-9ebe-34c15f68bc9d 13 2020-03-09T00:24:15.786 INFO:teuthology.orchestra.run.smithi104.stdout:13 2020-03-09T00:24:15.797 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 1ba69402-6d5c-491e-ac86-747517533a8b 14 2020-03-09T00:24:16.067 INFO:teuthology.orchestra.run.smithi104.stdout:14 2020-03-09T00:24:16.078 INFO:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new a3566380-0c4a-403f-8533-b2d0dcfa8506 15 2020-03-09T00:24:16.349 INFO:teuthology.orchestra.run.smithi104.stdout:15 2020-03-09T00:24:16.360 INFO:tasks.ceph.osd.2:Restarting daemon 2020-03-09T00:24:16.360 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-03-09T00:24:16.363 INFO:tasks.ceph.osd.2:Started 2020-03-09T00:24:16.363 INFO:tasks.ceph.osd.6:Restarting daemon 2020-03-09T00:24:16.363 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2020-03-09T00:24:16.366 INFO:tasks.ceph.osd.6:Started 2020-03-09T00:24:16.366 INFO:tasks.ceph.osd.10:Restarting daemon 2020-03-09T00:24:16.366 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2020-03-09T00:24:16.368 INFO:tasks.ceph.osd.10:Started 2020-03-09T00:24:16.368 INFO:tasks.ceph.osd.14:Restarting daemon 2020-03-09T00:24:16.368 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2020-03-09T00:24:16.370 INFO:tasks.ceph.osd.14:Started 2020-03-09T00:24:16.371 INFO:tasks.ceph.osd.0:Restarting daemon 2020-03-09T00:24:16.371 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-03-09T00:24:16.373 INFO:tasks.ceph.osd.0:Started 2020-03-09T00:24:16.373 INFO:tasks.ceph.osd.4:Restarting daemon 2020-03-09T00:24:16.374 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2020-03-09T00:24:16.376 INFO:tasks.ceph.osd.4:Started 2020-03-09T00:24:16.376 INFO:tasks.ceph.osd.8:Restarting daemon 2020-03-09T00:24:16.376 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2020-03-09T00:24:16.378 INFO:tasks.ceph.osd.8:Started 2020-03-09T00:24:16.378 INFO:tasks.ceph.osd.12:Restarting daemon 2020-03-09T00:24:16.379 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2020-03-09T00:24:16.381 INFO:tasks.ceph.osd.12:Started 2020-03-09T00:24:16.381 INFO:tasks.ceph.osd.1:Restarting daemon 2020-03-09T00:24:16.381 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-03-09T00:24:16.383 INFO:tasks.ceph.osd.1:Started 2020-03-09T00:24:16.384 INFO:tasks.ceph.osd.5:Restarting daemon 2020-03-09T00:24:16.384 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2020-03-09T00:24:16.386 INFO:tasks.ceph.osd.5:Started 2020-03-09T00:24:16.386 INFO:tasks.ceph.osd.9:Restarting daemon 2020-03-09T00:24:16.386 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2020-03-09T00:24:16.388 INFO:tasks.ceph.osd.9:Started 2020-03-09T00:24:16.389 INFO:tasks.ceph.osd.13:Restarting daemon 2020-03-09T00:24:16.389 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2020-03-09T00:24:16.391 INFO:tasks.ceph.osd.13:Started 2020-03-09T00:24:16.391 INFO:tasks.ceph.osd.3:Restarting daemon 2020-03-09T00:24:16.391 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2020-03-09T00:24:16.393 INFO:tasks.ceph.osd.3:Started 2020-03-09T00:24:16.394 INFO:tasks.ceph.osd.7:Restarting daemon 2020-03-09T00:24:16.394 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-03-09T00:24:16.396 INFO:tasks.ceph.osd.7:Started 2020-03-09T00:24:16.396 INFO:tasks.ceph.osd.11:Restarting daemon 2020-03-09T00:24:16.397 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2020-03-09T00:24:16.400 INFO:tasks.ceph.osd.11:Started 2020-03-09T00:24:16.400 INFO:tasks.ceph.osd.15:Restarting daemon 2020-03-09T00:24:16.400 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2020-03-09T00:24:16.402 INFO:tasks.ceph.osd.15:Started 2020-03-09T00:24:16.402 INFO:tasks.ceph:Waiting for OSDs to come up 2020-03-09T00:24:16.973 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:16.971+0000 7f8b018acec0 -1 Falling back to public interface 2020-03-09T00:24:16.977 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:16.975+0000 7f867ab5dec0 -1 Falling back to public interface 2020-03-09T00:24:16.977 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:16.975+0000 7f46573ccec0 -1 Falling back to public interface 2020-03-09T00:24:16.979 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:16.978+0000 7f8b3e896ec0 -1 Falling back to public interface 2020-03-09T00:24:16.980 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:16.978+0000 7f4651e31700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2020-03-09T00:24:16.986 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:16.984+0000 7efdb1f26ec0 -1 Falling back to public interface 2020-03-09T00:24:16.987 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:16.984+0000 7fd6d093fec0 -1 Falling back to public interface 2020-03-09T00:24:16.994 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:16.992+0000 7efc3e524ec0 -1 Falling back to public interface 2020-03-09T00:24:17.000 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:16.999+0000 7fbc72eeaec0 -1 Falling back to public interface 2020-03-09T00:24:17.001 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:16.998+0000 7fb704d77ec0 -1 Falling back to public interface 2020-03-09T00:24:17.004 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:17.002+0000 7fa1f02b4ec0 -1 Falling back to public interface 2020-03-09T00:24:17.009 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:17.007+0000 7f7b62f1aec0 -1 Falling back to public interface 2020-03-09T00:24:17.013 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:17.011+0000 7f4435cceec0 -1 Falling back to public interface 2020-03-09T00:24:17.015 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:17.013+0000 7f81bb83eec0 -1 Falling back to public interface 2020-03-09T00:24:17.016 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:17.014+0000 7f2560ca0ec0 -1 Falling back to public interface 2020-03-09T00:24:17.016 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:17.015+0000 7f1c64d44ec0 -1 Falling back to public interface 2020-03-09T00:24:17.020 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:17.019+0000 7f8563d35ec0 -1 Falling back to public interface 2020-03-09T00:24:17.803 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:17.802+0000 7f867ab5dec0 -1 osd.10 0 log_to_monitors {default=true} 2020-03-09T00:24:18.010 INFO:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-03-09T00:24:18.034 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:18.033+0000 7f8b018acec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-09T00:24:18.045 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:18.043+0000 7efdb1f26ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-09T00:24:18.050 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:18.048+0000 7f46573ccec0 -1 osd.6 0 log_to_monitors {default=true} 2020-03-09T00:24:18.052 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:18.051+0000 7fd6d093fec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-09T00:24:18.056 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:18.054+0000 7f8b3e896ec0 -1 osd.14 0 log_to_monitors {default=true} 2020-03-09T00:24:18.067 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:18.066+0000 7efc3e524ec0 -1 osd.8 0 log_to_monitors {default=true} 2020-03-09T00:24:18.070 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:18.069+0000 7fbc72eeaec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-09T00:24:18.080 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:18.078+0000 7fb704d77ec0 -1 osd.11 0 log_to_monitors {default=true} 2020-03-09T00:24:18.083 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:18.082+0000 7f7b62f1aec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-09T00:24:18.086 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:18.084+0000 7fa1f02b4ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-09T00:24:18.092 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:18.090+0000 7f4435cceec0 -1 osd.12 0 log_to_monitors {default=true} 2020-03-09T00:24:18.092 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:18.091+0000 7f81bb83eec0 -1 osd.7 0 log_to_monitors {default=true} 2020-03-09T00:24:18.092 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:18.091+0000 7f2560ca0ec0 -1 osd.15 0 log_to_monitors {default=true} 2020-03-09T00:24:18.096 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:18.094+0000 7f1c64d44ec0 -1 osd.13 0 log_to_monitors {default=true} 2020-03-09T00:24:18.098 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:18.097+0000 7f8563d35ec0 -1 osd.9 0 log_to_monitors {default=true} 2020-03-09T00:24:18.266 INFO:teuthology.misc.health.smithi148.stdout: 2020-03-09T00:24:18.266 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":21,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:24:17.856491+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"4","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:24:18.276 DEBUG:teuthology.misc:0 of 16 OSDs are up 2020-03-09T00:24:18.874 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:18.872+0000 7f865f307700 -1 osd.10 0 waiting for initial osdmap 2020-03-09T00:24:19.873 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:19.872+0000 7fb6e9521700 -1 osd.11 0 waiting for initial osdmap 2020-03-09T00:24:19.874 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:19.872+0000 7f8b23040700 -1 osd.14 0 waiting for initial osdmap 2020-03-09T00:24:19.874 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:19.872+0000 7f7b476c4700 -1 osd.0 0 waiting for initial osdmap 2020-03-09T00:24:19.875 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:19.873+0000 7f254544a700 -1 osd.15 0 waiting for initial osdmap 2020-03-09T00:24:19.875 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:19.874+0000 7f463bb76700 -1 osd.6 0 waiting for initial osdmap 2020-03-09T00:24:19.877 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:19.874+0000 7efd966d0700 -1 osd.3 0 waiting for initial osdmap 2020-03-09T00:24:19.880 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:19.879+0000 7f8ae6056700 -1 osd.2 0 waiting for initial osdmap 2020-03-09T00:24:19.880 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:19.878+0000 7f819ffe8700 -1 osd.7 0 waiting for initial osdmap 2020-03-09T00:24:19.881 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:19.879+0000 7fbc57694700 -1 osd.5 0 waiting for initial osdmap 2020-03-09T00:24:19.881 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:19.879+0000 7f1c494ee700 -1 osd.13 0 waiting for initial osdmap 2020-03-09T00:24:19.882 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:19.880+0000 7fa1d4a5e700 -1 osd.1 0 waiting for initial osdmap 2020-03-09T00:24:19.882 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:19.880+0000 7f85484df700 -1 osd.9 0 waiting for initial osdmap 2020-03-09T00:24:19.883 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:19.880+0000 7fd6b50e9700 -1 osd.4 0 waiting for initial osdmap 2020-03-09T00:24:19.883 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:19.880+0000 7efc22cce700 -1 osd.8 0 waiting for initial osdmap 2020-03-09T00:24:19.883 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:19.880+0000 7f441a478700 -1 osd.12 0 waiting for initial osdmap 2020-03-09T00:24:25.884 INFO:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-03-09T00:24:26.140 INFO:teuthology.misc.health.smithi148.stdout: 2020-03-09T00:24:26.141 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":26,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:24:22.888046+0000","last_up_change":"2020-03-09T00:24:20.875900+0000","last_in_change":"2020-03-09T00:24:20.875900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists","up"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111969+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.060238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.999353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.026007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.046589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111926+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.072146+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.062486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.773795+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.033423+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.138144+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.118617+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.039130+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.098144+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:24:26.151 DEBUG:teuthology.misc:16 of 16 OSDs are up 2020-03-09T00:24:26.151 INFO:tasks.ceph:Creating RBD pool 2020-03-09T00:24:26.152 INFO:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd pool create rbd 8 2020-03-09T00:24:26.914 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'rbd' created 2020-03-09T00:24:26.924 INFO:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-03-09T00:24:27.920 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rbd' on pool 'rbd' 2020-03-09T00:24:27.931 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-03-09T00:24:27.932 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:24:27.932 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-03-09T00:24:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:28.200 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:24:27.909920+0000","last_up_change":"2020-03-09T00:24:20.875900+0000","last_in_change":"2020-03-09T00:24:20.875900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists","up"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111969+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.060238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.999353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.026007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.046589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111926+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.072146+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.062486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.773795+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.033423+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.138144+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.118617+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.039130+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.098144+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:24:28.211 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 1, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:09.667386+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 1, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 1, u'pg_num': 1, u'type': 1, u'grade_table': [], u'pool_name': u'device_health_metrics', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 1, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'mgr_devicehealth': {}}, u'options': {u'pg_num_min': 1}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'26', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:26.419136+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'28', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-03-09T00:24:28.212 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool get device_health_metrics pg_num 2020-03-09T00:24:28.469 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2020-03-09T00:24:28.480 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool get rbd pg_num 2020-03-09T00:24:28.740 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 8 2020-03-09T00:24:28.751 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-03-09T00:24:28.751 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-03-09T00:24:28.752 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr dump --format=json 2020-03-09T00:24:29.047 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:29.058 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":4,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":26774},{"type":"v1","addr":"172.21.15.104:6801","nonce":26774}]},"active_addr":"172.21.15.104:6801/26774","active_change":"2020-03-09T00:24:09.641716+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":579184066}]},{"addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":3476624431}]},{"addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":1512436969}]}]}} 2020-03-09T00:24:29.063 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-03-09T00:24:29.063 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-03-09T00:24:29.063 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:24:29.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:29.329 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":29,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:24:28.913404+0000","last_up_change":"2020-03-09T00:24:20.875900+0000","last_in_change":"2020-03-09T00:24:20.875900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists","up"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111969+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.060238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.999353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.026007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.046589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111926+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.072146+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.062486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.773795+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.033423+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.138144+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.118617+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.039130+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.098144+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:24:29.341 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-03-09T00:24:29.341 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:24:29.599 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:29.599 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":29,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:24:28.913404+0000","last_up_change":"2020-03-09T00:24:20.875900+0000","last_in_change":"2020-03-09T00:24:20.875900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists","up"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111969+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.060238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.999353+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.026007+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.046589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.111926+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.072146+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.062486+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:18.773795+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.033423+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.138144+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.118617+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.039130+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.098144+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:24:29.611 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.0 flush_pg_stats 2020-03-09T00:24:29.753 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:29.754 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.1 flush_pg_stats 2020-03-09T00:24:29.896 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:29.896 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 flush_pg_stats 2020-03-09T00:24:30.038 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:30.038 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.3 flush_pg_stats 2020-03-09T00:24:30.180 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:30.180 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.4 flush_pg_stats 2020-03-09T00:24:30.322 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:30.322 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.5 flush_pg_stats 2020-03-09T00:24:30.465 INFO:teuthology.orchestra.run.smithi148.stdout:103079215109 2020-03-09T00:24:30.465 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.6 flush_pg_stats 2020-03-09T00:24:30.607 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:30.608 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.7 flush_pg_stats 2020-03-09T00:24:30.747 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:30.748 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.8 flush_pg_stats 2020-03-09T00:24:30.888 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:30.889 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.9 flush_pg_stats 2020-03-09T00:24:31.030 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:31.030 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.10 flush_pg_stats 2020-03-09T00:24:31.172 INFO:teuthology.orchestra.run.smithi148.stdout:98784247813 2020-03-09T00:24:31.173 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.11 flush_pg_stats 2020-03-09T00:24:31.313 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:31.313 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.12 flush_pg_stats 2020-03-09T00:24:31.454 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:31.455 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.13 flush_pg_stats 2020-03-09T00:24:31.596 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:31.596 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.14 flush_pg_stats 2020-03-09T00:24:31.738 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:31.738 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.15 flush_pg_stats 2020-03-09T00:24:31.881 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:31.881 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.0 2020-03-09T00:24:32.138 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:32.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215107 for osd.0 2020-03-09T00:24:32.150 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.1 2020-03-09T00:24:32.405 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:32.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215107 for osd.1 2020-03-09T00:24:32.417 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.2 2020-03-09T00:24:32.673 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:32.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215107 for osd.2 2020-03-09T00:24:32.685 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.3 2020-03-09T00:24:32.942 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:32.953 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215108 for osd.3 2020-03-09T00:24:32.953 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.4 2020-03-09T00:24:33.211 INFO:teuthology.orchestra.run.smithi148.stdout:103079215107 2020-03-09T00:24:33.222 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215107 for osd.4 2020-03-09T00:24:33.222 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.5 2020-03-09T00:24:33.480 INFO:teuthology.orchestra.run.smithi148.stdout:103079215109 2020-03-09T00:24:33.491 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215109 got 103079215109 for osd.5 2020-03-09T00:24:33.492 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.6 2020-03-09T00:24:33.749 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:33.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215108 for osd.6 2020-03-09T00:24:33.760 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.7 2020-03-09T00:24:34.015 INFO:teuthology.orchestra.run.smithi148.stdout:103079215109 2020-03-09T00:24:34.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215109 for osd.7 2020-03-09T00:24:34.027 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.8 2020-03-09T00:24:34.282 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:34.293 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215108 for osd.8 2020-03-09T00:24:34.294 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.9 2020-03-09T00:24:34.550 INFO:teuthology.orchestra.run.smithi148.stdout:103079215109 2020-03-09T00:24:34.562 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215109 for osd.9 2020-03-09T00:24:34.562 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.10 2020-03-09T00:24:34.818 INFO:teuthology.orchestra.run.smithi148.stdout:98784247813 2020-03-09T00:24:34.830 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.10 2020-03-09T00:24:34.830 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.11 2020-03-09T00:24:35.089 INFO:teuthology.orchestra.run.smithi148.stdout:103079215109 2020-03-09T00:24:35.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215109 for osd.11 2020-03-09T00:24:35.101 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.12 2020-03-09T00:24:35.357 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:35.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215108 for osd.12 2020-03-09T00:24:35.369 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.13 2020-03-09T00:24:35.627 INFO:teuthology.orchestra.run.smithi148.stdout:103079215109 2020-03-09T00:24:35.638 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215108 got 103079215109 for osd.13 2020-03-09T00:24:35.638 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.14 2020-03-09T00:24:35.895 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:35.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215108 for osd.14 2020-03-09T00:24:35.906 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd last-stat-seq osd.15 2020-03-09T00:24:36.165 INFO:teuthology.orchestra.run.smithi148.stdout:103079215108 2020-03-09T00:24:36.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 103079215107 got 103079215108 for osd.15 2020-03-09T00:24:36.177 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-03-09T00:24:36.177 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-03-09T00:24:36.431 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:36.432 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2020-03-09T00:24:36.442 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2020-03-09T00:24:35.649167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":16781696,"kb_used_data":4352,"kb_used_omap":0,"kb_used_meta":16777216,"kb_avail":1493167744,"statfs":{"total":1546188226560,"available":1529003769856,"internally_reserved":17179869184,"allocated":4456448,"data_stored":1939136,"data_compressed":174720,"data_compressed_allocated":393216,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":17179869184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002079"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919758+0000","last_change":"2020-03-09T00:24:27.919488+0000","last_active":"2020-03-09T00:24:27.919758+0000","last_peered":"2020-03-09T00:24:27.919758+0000","last_clean":"2020-03-09T00:24:27.919758+0000","last_became_active":"2020-03-09T00:24:27.919102+0000","last_became_peered":"2020-03-09T00:24:27.919102+0000","last_unstale":"2020-03-09T00:24:27.919758+0000","last_undegraded":"2020-03-09T00:24:27.919758+0000","last_fullsized":"2020-03-09T00:24:27.919758+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941734+0000","last_change":"2020-03-09T00:24:27.941155+0000","last_active":"2020-03-09T00:24:27.941734+0000","last_peered":"2020-03-09T00:24:27.941734+0000","last_clean":"2020-03-09T00:24:27.941734+0000","last_became_active":"2020-03-09T00:24:27.940788+0000","last_became_peered":"2020-03-09T00:24:27.940788+0000","last_unstale":"2020-03-09T00:24:27.941734+0000","last_undegraded":"2020-03-09T00:24:27.941734+0000","last_fullsized":"2020-03-09T00:24:27.941734+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.925764+0000","last_change":"2020-03-09T00:24:27.925113+0000","last_active":"2020-03-09T00:24:27.925764+0000","last_peered":"2020-03-09T00:24:27.925764+0000","last_clean":"2020-03-09T00:24:27.925764+0000","last_became_active":"2020-03-09T00:24:27.924718+0000","last_became_peered":"2020-03-09T00:24:27.924718+0000","last_unstale":"2020-03-09T00:24:27.925764+0000","last_undegraded":"2020-03-09T00:24:27.925764+0000","last_fullsized":"2020-03-09T00:24:27.925764+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941598+0000","last_change":"2020-03-09T00:24:27.941291+0000","last_active":"2020-03-09T00:24:27.941598+0000","last_peered":"2020-03-09T00:24:27.941598+0000","last_clean":"2020-03-09T00:24:27.941598+0000","last_became_active":"2020-03-09T00:24:27.940904+0000","last_became_peered":"2020-03-09T00:24:27.940904+0000","last_unstale":"2020-03-09T00:24:27.941598+0000","last_undegraded":"2020-03-09T00:24:27.941598+0000","last_fullsized":"2020-03-09T00:24:27.941598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"26'3","reported_seq":"15","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.917436+0000","last_change":"2020-03-09T00:24:22.183631+0000","last_active":"2020-03-09T00:24:27.917436+0000","last_peered":"2020-03-09T00:24:27.917436+0000","last_clean":"2020-03-09T00:24:27.917436+0000","last_became_active":"2020-03-09T00:24:22.183150+0000","last_became_peered":"2020-03-09T00:24:22.183150+0000","last_unstale":"2020-03-09T00:24:27.917436+0000","last_undegraded":"2020-03-09T00:24:27.917436+0000","last_fullsized":"2020-03-09T00:24:27.917436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_clean_scrub_stamp":"2020-03-09T00:24:10.644148+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919286+0000","last_change":"2020-03-09T00:24:27.919033+0000","last_active":"2020-03-09T00:24:27.919286+0000","last_peered":"2020-03-09T00:24:27.919286+0000","last_clean":"2020-03-09T00:24:27.919286+0000","last_became_active":"2020-03-09T00:24:27.918573+0000","last_became_peered":"2020-03-09T00:24:27.918573+0000","last_unstale":"2020-03-09T00:24:27.919286+0000","last_undegraded":"2020-03-09T00:24:27.919286+0000","last_fullsized":"2020-03-09T00:24:27.919286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.920549+0000","last_change":"2020-03-09T00:24:27.920285+0000","last_active":"2020-03-09T00:24:27.920549+0000","last_peered":"2020-03-09T00:24:27.920549+0000","last_clean":"2020-03-09T00:24:27.920549+0000","last_became_active":"2020-03-09T00:24:27.919842+0000","last_became_peered":"2020-03-09T00:24:27.919842+0000","last_unstale":"2020-03-09T00:24:27.920549+0000","last_undegraded":"2020-03-09T00:24:27.920549+0000","last_fullsized":"2020-03-09T00:24:27.920549+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.942680+0000","last_change":"2020-03-09T00:24:27.942443+0000","last_active":"2020-03-09T00:24:27.942680+0000","last_peered":"2020-03-09T00:24:27.942680+0000","last_clean":"2020-03-09T00:24:27.942680+0000","last_became_active":"2020-03-09T00:24:27.942050+0000","last_became_peered":"2020-03-09T00:24:27.942050+0000","last_unstale":"2020-03-09T00:24:27.942680+0000","last_undegraded":"2020-03-09T00:24:27.942680+0000","last_fullsized":"2020-03-09T00:24:27.942680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:28.136197+0000","last_change":"2020-03-09T00:24:28.135611+0000","last_active":"2020-03-09T00:24:28.136197+0000","last_peered":"2020-03-09T00:24:28.136197+0000","last_clean":"2020-03-09T00:24:28.136197+0000","last_became_active":"2020-03-09T00:24:28.135155+0000","last_became_peered":"2020-03-09T00:24:28.135155+0000","last_unstale":"2020-03-09T00:24:28.136197+0000","last_undegraded":"2020-03-09T00:24:28.136197+0000","last_fullsized":"2020-03-09T00:24:28.136197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12],"acting":[6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":24,"seq":103079215108,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2669999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7789999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1160000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]}]},{"osd":14,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.722}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":13,"up_from":24,"seq":103079215109,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]}]},{"osd":12,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]}]},{"osd":11,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":9,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":8,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]}]},{"osd":7,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]}]},{"osd":6,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]}]},{"osd":5,"up_from":24,"seq":103079215110,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215108,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8109999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]}]},{"osd":10,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]}]},{"osd":3,"up_from":24,"seq":103079215109,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.841}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]}]},{"osd":1,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6930000000000001}]}]},{"osd":2,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.331}]}]},{"osd":4,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]}]}]}} 2020-03-09T00:24:36.446 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-03-09T00:24:36.702 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:36.703 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2020-03-09T00:24:36.713 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2020-03-09T00:24:35.649167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":16781696,"kb_used_data":4352,"kb_used_omap":0,"kb_used_meta":16777216,"kb_avail":1493167744,"statfs":{"total":1546188226560,"available":1529003769856,"internally_reserved":17179869184,"allocated":4456448,"data_stored":1939136,"data_compressed":174720,"data_compressed_allocated":393216,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":17179869184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002079"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919758+0000","last_change":"2020-03-09T00:24:27.919488+0000","last_active":"2020-03-09T00:24:27.919758+0000","last_peered":"2020-03-09T00:24:27.919758+0000","last_clean":"2020-03-09T00:24:27.919758+0000","last_became_active":"2020-03-09T00:24:27.919102+0000","last_became_peered":"2020-03-09T00:24:27.919102+0000","last_unstale":"2020-03-09T00:24:27.919758+0000","last_undegraded":"2020-03-09T00:24:27.919758+0000","last_fullsized":"2020-03-09T00:24:27.919758+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941734+0000","last_change":"2020-03-09T00:24:27.941155+0000","last_active":"2020-03-09T00:24:27.941734+0000","last_peered":"2020-03-09T00:24:27.941734+0000","last_clean":"2020-03-09T00:24:27.941734+0000","last_became_active":"2020-03-09T00:24:27.940788+0000","last_became_peered":"2020-03-09T00:24:27.940788+0000","last_unstale":"2020-03-09T00:24:27.941734+0000","last_undegraded":"2020-03-09T00:24:27.941734+0000","last_fullsized":"2020-03-09T00:24:27.941734+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.925764+0000","last_change":"2020-03-09T00:24:27.925113+0000","last_active":"2020-03-09T00:24:27.925764+0000","last_peered":"2020-03-09T00:24:27.925764+0000","last_clean":"2020-03-09T00:24:27.925764+0000","last_became_active":"2020-03-09T00:24:27.924718+0000","last_became_peered":"2020-03-09T00:24:27.924718+0000","last_unstale":"2020-03-09T00:24:27.925764+0000","last_undegraded":"2020-03-09T00:24:27.925764+0000","last_fullsized":"2020-03-09T00:24:27.925764+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941598+0000","last_change":"2020-03-09T00:24:27.941291+0000","last_active":"2020-03-09T00:24:27.941598+0000","last_peered":"2020-03-09T00:24:27.941598+0000","last_clean":"2020-03-09T00:24:27.941598+0000","last_became_active":"2020-03-09T00:24:27.940904+0000","last_became_peered":"2020-03-09T00:24:27.940904+0000","last_unstale":"2020-03-09T00:24:27.941598+0000","last_undegraded":"2020-03-09T00:24:27.941598+0000","last_fullsized":"2020-03-09T00:24:27.941598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"26'3","reported_seq":"15","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.917436+0000","last_change":"2020-03-09T00:24:22.183631+0000","last_active":"2020-03-09T00:24:27.917436+0000","last_peered":"2020-03-09T00:24:27.917436+0000","last_clean":"2020-03-09T00:24:27.917436+0000","last_became_active":"2020-03-09T00:24:22.183150+0000","last_became_peered":"2020-03-09T00:24:22.183150+0000","last_unstale":"2020-03-09T00:24:27.917436+0000","last_undegraded":"2020-03-09T00:24:27.917436+0000","last_fullsized":"2020-03-09T00:24:27.917436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_clean_scrub_stamp":"2020-03-09T00:24:10.644148+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919286+0000","last_change":"2020-03-09T00:24:27.919033+0000","last_active":"2020-03-09T00:24:27.919286+0000","last_peered":"2020-03-09T00:24:27.919286+0000","last_clean":"2020-03-09T00:24:27.919286+0000","last_became_active":"2020-03-09T00:24:27.918573+0000","last_became_peered":"2020-03-09T00:24:27.918573+0000","last_unstale":"2020-03-09T00:24:27.919286+0000","last_undegraded":"2020-03-09T00:24:27.919286+0000","last_fullsized":"2020-03-09T00:24:27.919286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.920549+0000","last_change":"2020-03-09T00:24:27.920285+0000","last_active":"2020-03-09T00:24:27.920549+0000","last_peered":"2020-03-09T00:24:27.920549+0000","last_clean":"2020-03-09T00:24:27.920549+0000","last_became_active":"2020-03-09T00:24:27.919842+0000","last_became_peered":"2020-03-09T00:24:27.919842+0000","last_unstale":"2020-03-09T00:24:27.920549+0000","last_undegraded":"2020-03-09T00:24:27.920549+0000","last_fullsized":"2020-03-09T00:24:27.920549+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.942680+0000","last_change":"2020-03-09T00:24:27.942443+0000","last_active":"2020-03-09T00:24:27.942680+0000","last_peered":"2020-03-09T00:24:27.942680+0000","last_clean":"2020-03-09T00:24:27.942680+0000","last_became_active":"2020-03-09T00:24:27.942050+0000","last_became_peered":"2020-03-09T00:24:27.942050+0000","last_unstale":"2020-03-09T00:24:27.942680+0000","last_undegraded":"2020-03-09T00:24:27.942680+0000","last_fullsized":"2020-03-09T00:24:27.942680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:28.136197+0000","last_change":"2020-03-09T00:24:28.135611+0000","last_active":"2020-03-09T00:24:28.136197+0000","last_peered":"2020-03-09T00:24:28.136197+0000","last_clean":"2020-03-09T00:24:28.136197+0000","last_became_active":"2020-03-09T00:24:28.135155+0000","last_became_peered":"2020-03-09T00:24:28.135155+0000","last_unstale":"2020-03-09T00:24:28.136197+0000","last_undegraded":"2020-03-09T00:24:28.136197+0000","last_fullsized":"2020-03-09T00:24:28.136197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12],"acting":[6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":24,"seq":103079215108,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2669999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7789999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1160000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]}]},{"osd":14,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.722}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":13,"up_from":24,"seq":103079215109,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]}]},{"osd":12,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]}]},{"osd":11,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":9,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":8,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]}]},{"osd":7,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]}]},{"osd":6,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]}]},{"osd":5,"up_from":24,"seq":103079215110,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215108,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8109999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]}]},{"osd":10,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]}]},{"osd":3,"up_from":24,"seq":103079215109,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.841}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]}]},{"osd":1,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6930000000000001}]}]},{"osd":2,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.331}]}]},{"osd":4,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]}]}]}} 2020-03-09T00:24:36.719 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-03-09T00:24:36.719 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-03-09T00:24:36.719 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-03-09T00:24:36.720 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early health --format=json 2020-03-09T00:24:37.025 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:37.036 INFO:teuthology.orchestra.run.smithi148.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-03-09T00:24:37.037 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-03-09T00:24:37.037 INFO:teuthology.run_tasks:Running task thrashosds... 2020-03-09T00:24:37.047 INFO:tasks.thrashosds:config is {'dump_ops_enable': 'true', 'random_eio': 0.0, 'optrack_toggle_delay': 2.0, 'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'noscrub_toggle_delay': 2.0, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'min_in': 8, 'sighup_delay': 0.1, 'chance_pgnum_shrink': 1} 2020-03-09T00:24:37.047 INFO:tasks.thrashosds:overrides is {'ceph': {'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'conf': {'global': {'ms tcp read timeout': 5, 'ms inject socket failures': 2500, 'mon client directed command retry': 5, 'osd_async_recovery_min_cost': 1}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'mon osd backfillfull_ratio': 0.85, 'debug journal': 20, 'osd scrub max interval': 120, 'bluestore compression algorithm': 'zlib', 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug verify missing on start': True, 'debug ms': 20, 'osd debug reject backfill probability': 0.3, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'bluestore fsck on mount': True, 'osd max backfills': 3, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd debug verify cached snaps': True, 'osd max markdown count': 1000, 'osd op queue': 'debug_random', 'bluestore block size': 96636764160, 'osd scrub min interval': 60, 'debug filestore': 20, 'osd failsafe full ratio': 0.95}}, 'fs': 'xfs', 'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b'}}, 'admin_socket': {'branch': 'wip-sage4-testing-2020-03-08-0938'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'sha1': '9eb9490ed0531fd33292066429bf9db55b9c1d3b', 'branch': 'wip-sage4-testing-2020-03-08-0938'}} 2020-03-09T00:24:37.048 INFO:tasks.thrashosds:config is {'dump_ops_enable': 'true', 'random_eio': 0.0, 'bdev_inject_crash_probability': 0.5, 'optrack_toggle_delay': 2.0, 'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'bdev_inject_crash': 2, 'noscrub_toggle_delay': 2.0, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'min_in': 8, 'sighup_delay': 0.1, 'chance_pgnum_shrink': 1} 2020-03-09T00:24:37.048 INFO:tasks.thrashosds:Beginning thrashosds... 2020-03-09T00:24:37.048 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-03-09T00:24:37.048 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-03-09T00:24:37.304 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:37.305 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2020-03-09T00:24:37.315 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2020-03-09T00:24:35.649167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":16781696,"kb_used_data":4352,"kb_used_omap":0,"kb_used_meta":16777216,"kb_avail":1493167744,"statfs":{"total":1546188226560,"available":1529003769856,"internally_reserved":17179869184,"allocated":4456448,"data_stored":1939136,"data_compressed":174720,"data_compressed_allocated":393216,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":17179869184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002079"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919758+0000","last_change":"2020-03-09T00:24:27.919488+0000","last_active":"2020-03-09T00:24:27.919758+0000","last_peered":"2020-03-09T00:24:27.919758+0000","last_clean":"2020-03-09T00:24:27.919758+0000","last_became_active":"2020-03-09T00:24:27.919102+0000","last_became_peered":"2020-03-09T00:24:27.919102+0000","last_unstale":"2020-03-09T00:24:27.919758+0000","last_undegraded":"2020-03-09T00:24:27.919758+0000","last_fullsized":"2020-03-09T00:24:27.919758+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941734+0000","last_change":"2020-03-09T00:24:27.941155+0000","last_active":"2020-03-09T00:24:27.941734+0000","last_peered":"2020-03-09T00:24:27.941734+0000","last_clean":"2020-03-09T00:24:27.941734+0000","last_became_active":"2020-03-09T00:24:27.940788+0000","last_became_peered":"2020-03-09T00:24:27.940788+0000","last_unstale":"2020-03-09T00:24:27.941734+0000","last_undegraded":"2020-03-09T00:24:27.941734+0000","last_fullsized":"2020-03-09T00:24:27.941734+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.925764+0000","last_change":"2020-03-09T00:24:27.925113+0000","last_active":"2020-03-09T00:24:27.925764+0000","last_peered":"2020-03-09T00:24:27.925764+0000","last_clean":"2020-03-09T00:24:27.925764+0000","last_became_active":"2020-03-09T00:24:27.924718+0000","last_became_peered":"2020-03-09T00:24:27.924718+0000","last_unstale":"2020-03-09T00:24:27.925764+0000","last_undegraded":"2020-03-09T00:24:27.925764+0000","last_fullsized":"2020-03-09T00:24:27.925764+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941598+0000","last_change":"2020-03-09T00:24:27.941291+0000","last_active":"2020-03-09T00:24:27.941598+0000","last_peered":"2020-03-09T00:24:27.941598+0000","last_clean":"2020-03-09T00:24:27.941598+0000","last_became_active":"2020-03-09T00:24:27.940904+0000","last_became_peered":"2020-03-09T00:24:27.940904+0000","last_unstale":"2020-03-09T00:24:27.941598+0000","last_undegraded":"2020-03-09T00:24:27.941598+0000","last_fullsized":"2020-03-09T00:24:27.941598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"26'3","reported_seq":"15","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.917436+0000","last_change":"2020-03-09T00:24:22.183631+0000","last_active":"2020-03-09T00:24:27.917436+0000","last_peered":"2020-03-09T00:24:27.917436+0000","last_clean":"2020-03-09T00:24:27.917436+0000","last_became_active":"2020-03-09T00:24:22.183150+0000","last_became_peered":"2020-03-09T00:24:22.183150+0000","last_unstale":"2020-03-09T00:24:27.917436+0000","last_undegraded":"2020-03-09T00:24:27.917436+0000","last_fullsized":"2020-03-09T00:24:27.917436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_clean_scrub_stamp":"2020-03-09T00:24:10.644148+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919286+0000","last_change":"2020-03-09T00:24:27.919033+0000","last_active":"2020-03-09T00:24:27.919286+0000","last_peered":"2020-03-09T00:24:27.919286+0000","last_clean":"2020-03-09T00:24:27.919286+0000","last_became_active":"2020-03-09T00:24:27.918573+0000","last_became_peered":"2020-03-09T00:24:27.918573+0000","last_unstale":"2020-03-09T00:24:27.919286+0000","last_undegraded":"2020-03-09T00:24:27.919286+0000","last_fullsized":"2020-03-09T00:24:27.919286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.920549+0000","last_change":"2020-03-09T00:24:27.920285+0000","last_active":"2020-03-09T00:24:27.920549+0000","last_peered":"2020-03-09T00:24:27.920549+0000","last_clean":"2020-03-09T00:24:27.920549+0000","last_became_active":"2020-03-09T00:24:27.919842+0000","last_became_peered":"2020-03-09T00:24:27.919842+0000","last_unstale":"2020-03-09T00:24:27.920549+0000","last_undegraded":"2020-03-09T00:24:27.920549+0000","last_fullsized":"2020-03-09T00:24:27.920549+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.942680+0000","last_change":"2020-03-09T00:24:27.942443+0000","last_active":"2020-03-09T00:24:27.942680+0000","last_peered":"2020-03-09T00:24:27.942680+0000","last_clean":"2020-03-09T00:24:27.942680+0000","last_became_active":"2020-03-09T00:24:27.942050+0000","last_became_peered":"2020-03-09T00:24:27.942050+0000","last_unstale":"2020-03-09T00:24:27.942680+0000","last_undegraded":"2020-03-09T00:24:27.942680+0000","last_fullsized":"2020-03-09T00:24:27.942680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:28.136197+0000","last_change":"2020-03-09T00:24:28.135611+0000","last_active":"2020-03-09T00:24:28.136197+0000","last_peered":"2020-03-09T00:24:28.136197+0000","last_clean":"2020-03-09T00:24:28.136197+0000","last_became_active":"2020-03-09T00:24:28.135155+0000","last_became_peered":"2020-03-09T00:24:28.135155+0000","last_unstale":"2020-03-09T00:24:28.136197+0000","last_undegraded":"2020-03-09T00:24:28.136197+0000","last_fullsized":"2020-03-09T00:24:28.136197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12],"acting":[6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":24,"seq":103079215108,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2669999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7789999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1160000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]}]},{"osd":14,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.722}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":13,"up_from":24,"seq":103079215109,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]}]},{"osd":12,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]}]},{"osd":11,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":9,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":8,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]}]},{"osd":7,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]}]},{"osd":6,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]}]},{"osd":5,"up_from":24,"seq":103079215110,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215108,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8109999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]}]},{"osd":10,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]}]},{"osd":3,"up_from":24,"seq":103079215109,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.841}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]}]},{"osd":1,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6930000000000001}]}]},{"osd":2,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.331}]}]},{"osd":4,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]}]}]}} 2020-03-09T00:24:37.351 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-03-09T00:24:37.605 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:37.606 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2020-03-09T00:24:37.616 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2020-03-09T00:24:35.649167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1509949440,"kb_used":16781696,"kb_used_data":4352,"kb_used_omap":0,"kb_used_meta":16777216,"kb_avail":1493167744,"statfs":{"total":1546188226560,"available":1529003769856,"internally_reserved":17179869184,"allocated":4456448,"data_stored":1939136,"data_compressed":174720,"data_compressed_allocated":393216,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":17179869184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002079"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919758+0000","last_change":"2020-03-09T00:24:27.919488+0000","last_active":"2020-03-09T00:24:27.919758+0000","last_peered":"2020-03-09T00:24:27.919758+0000","last_clean":"2020-03-09T00:24:27.919758+0000","last_became_active":"2020-03-09T00:24:27.919102+0000","last_became_peered":"2020-03-09T00:24:27.919102+0000","last_unstale":"2020-03-09T00:24:27.919758+0000","last_undegraded":"2020-03-09T00:24:27.919758+0000","last_fullsized":"2020-03-09T00:24:27.919758+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941734+0000","last_change":"2020-03-09T00:24:27.941155+0000","last_active":"2020-03-09T00:24:27.941734+0000","last_peered":"2020-03-09T00:24:27.941734+0000","last_clean":"2020-03-09T00:24:27.941734+0000","last_became_active":"2020-03-09T00:24:27.940788+0000","last_became_peered":"2020-03-09T00:24:27.940788+0000","last_unstale":"2020-03-09T00:24:27.941734+0000","last_undegraded":"2020-03-09T00:24:27.941734+0000","last_fullsized":"2020-03-09T00:24:27.941734+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.925764+0000","last_change":"2020-03-09T00:24:27.925113+0000","last_active":"2020-03-09T00:24:27.925764+0000","last_peered":"2020-03-09T00:24:27.925764+0000","last_clean":"2020-03-09T00:24:27.925764+0000","last_became_active":"2020-03-09T00:24:27.924718+0000","last_became_peered":"2020-03-09T00:24:27.924718+0000","last_unstale":"2020-03-09T00:24:27.925764+0000","last_undegraded":"2020-03-09T00:24:27.925764+0000","last_fullsized":"2020-03-09T00:24:27.925764+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.941598+0000","last_change":"2020-03-09T00:24:27.941291+0000","last_active":"2020-03-09T00:24:27.941598+0000","last_peered":"2020-03-09T00:24:27.941598+0000","last_clean":"2020-03-09T00:24:27.941598+0000","last_became_active":"2020-03-09T00:24:27.940904+0000","last_became_peered":"2020-03-09T00:24:27.940904+0000","last_unstale":"2020-03-09T00:24:27.941598+0000","last_undegraded":"2020-03-09T00:24:27.941598+0000","last_fullsized":"2020-03-09T00:24:27.941598+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"26'3","reported_seq":"15","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.917436+0000","last_change":"2020-03-09T00:24:22.183631+0000","last_active":"2020-03-09T00:24:27.917436+0000","last_peered":"2020-03-09T00:24:27.917436+0000","last_clean":"2020-03-09T00:24:27.917436+0000","last_became_active":"2020-03-09T00:24:22.183150+0000","last_became_peered":"2020-03-09T00:24:22.183150+0000","last_unstale":"2020-03-09T00:24:27.917436+0000","last_undegraded":"2020-03-09T00:24:27.917436+0000","last_fullsized":"2020-03-09T00:24:27.917436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_clean_scrub_stamp":"2020-03-09T00:24:10.644148+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.919286+0000","last_change":"2020-03-09T00:24:27.919033+0000","last_active":"2020-03-09T00:24:27.919286+0000","last_peered":"2020-03-09T00:24:27.919286+0000","last_clean":"2020-03-09T00:24:27.919286+0000","last_became_active":"2020-03-09T00:24:27.918573+0000","last_became_peered":"2020-03-09T00:24:27.918573+0000","last_unstale":"2020-03-09T00:24:27.919286+0000","last_undegraded":"2020-03-09T00:24:27.919286+0000","last_fullsized":"2020-03-09T00:24:27.919286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.920549+0000","last_change":"2020-03-09T00:24:27.920285+0000","last_active":"2020-03-09T00:24:27.920549+0000","last_peered":"2020-03-09T00:24:27.920549+0000","last_clean":"2020-03-09T00:24:27.920549+0000","last_became_active":"2020-03-09T00:24:27.919842+0000","last_became_peered":"2020-03-09T00:24:27.919842+0000","last_unstale":"2020-03-09T00:24:27.920549+0000","last_undegraded":"2020-03-09T00:24:27.920549+0000","last_fullsized":"2020-03-09T00:24:27.920549+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:27.942680+0000","last_change":"2020-03-09T00:24:27.942443+0000","last_active":"2020-03-09T00:24:27.942680+0000","last_peered":"2020-03-09T00:24:27.942680+0000","last_clean":"2020-03-09T00:24:27.942680+0000","last_became_active":"2020-03-09T00:24:27.942050+0000","last_became_peered":"2020-03-09T00:24:27.942050+0000","last_unstale":"2020-03-09T00:24:27.942680+0000","last_undegraded":"2020-03-09T00:24:27.942680+0000","last_fullsized":"2020-03-09T00:24:27.942680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"28","state":"active+clean","last_fresh":"2020-03-09T00:24:28.136197+0000","last_change":"2020-03-09T00:24:28.135611+0000","last_active":"2020-03-09T00:24:28.136197+0000","last_peered":"2020-03-09T00:24:28.136197+0000","last_clean":"2020-03-09T00:24:28.136197+0000","last_became_active":"2020-03-09T00:24:28.135155+0000","last_became_peered":"2020-03-09T00:24:28.135155+0000","last_unstale":"2020-03-09T00:24:28.136197+0000","last_undegraded":"2020-03-09T00:24:28.136197+0000","last_fullsized":"2020-03-09T00:24:28.136197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12],"acting":[6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":24,"seq":103079215108,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.677}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2669999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7789999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1160000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]}]},{"osd":14,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.722}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.706}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]}]},{"osd":13,"up_from":24,"seq":103079215109,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,5,6,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.069}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]}]},{"osd":12,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6799999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]}]},{"osd":11,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":9,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.976}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2}]}]},{"osd":8,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]}]},{"osd":7,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]}]},{"osd":6,"up_from":24,"seq":103079215109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]}]},{"osd":5,"up_from":24,"seq":103079215110,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[2,3,4,6,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6990000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215108,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.518}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8109999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]}]},{"osd":10,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]}]},{"osd":3,"up_from":24,"seq":103079215109,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.841}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]}]},{"osd":1,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6930000000000001}]}]},{"osd":2,"up_from":24,"seq":103079215108,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.331}]}]},{"osd":4,"up_from":24,"seq":103079215108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1048856,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322984,"statfs":{"total":96636764160,"available":95562735616,"internally_reserved":1073741824,"allocated":278528,"data_stored":121196,"data_compressed":10920,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]}]}]}} 2020-03-09T00:24:37.622 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-03-09T00:24:37.623 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump 2020-03-09T00:24:37.880 INFO:teuthology.orchestra.run.smithi148.stdout:epoch 29 2020-03-09T00:24:37.880 INFO:teuthology.orchestra.run.smithi148.stdout:fsid 036b76b6-511e-4ec6-9669-0a47550bd261 2020-03-09T00:24:37.881 INFO:teuthology.orchestra.run.smithi148.stdout:created 2020-03-09T00:24:06.736934+0000 2020-03-09T00:24:37.881 INFO:teuthology.orchestra.run.smithi148.stdout:modified 2020-03-09T00:24:28.913404+0000 2020-03-09T00:24:37.881 INFO:teuthology.orchestra.run.smithi148.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2020-03-09T00:24:37.881 INFO:teuthology.orchestra.run.smithi148.stdout:crush_version 5 2020-03-09T00:24:37.881 INFO:teuthology.orchestra.run.smithi148.stdout:full_ratio 0.95 2020-03-09T00:24:37.882 INFO:teuthology.orchestra.run.smithi148.stdout:backfillfull_ratio 0.9 2020-03-09T00:24:37.882 INFO:teuthology.orchestra.run.smithi148.stdout:nearfull_ratio 0.85 2020-03-09T00:24:37.882 INFO:teuthology.orchestra.run.smithi148.stdout:require_min_compat_client jewel 2020-03-09T00:24:37.882 INFO:teuthology.orchestra.run.smithi148.stdout:min_compat_client jewel 2020-03-09T00:24:37.882 INFO:teuthology.orchestra.run.smithi148.stdout:require_osd_release octopus 2020-03-09T00:24:37.882 INFO:teuthology.orchestra.run.smithi148.stdout:pool 1 'device_health_metrics' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 26 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth 2020-03-09T00:24:37.883 INFO:teuthology.orchestra.run.smithi148.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 29 flags hashpspool stripe_width 0 application rbd 2020-03-09T00:24:37.883 INFO:teuthology.orchestra.run.smithi148.stdout:max_osd 16 2020-03-09T00:24:37.883 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6816/27385,v1:172.21.15.148:6817/27385] [v2:172.21.15.148:6818/27385,v1:172.21.15.148:6819/27385] exists,up d0be76d1-3ce6-4c23-9917-41b35482def3 2020-03-09T00:24:37.883 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6808/27629,v1:172.21.15.50:6809/27629] [v2:172.21.15.50:6810/27629,v1:172.21.15.50:6811/27629] exists,up cee70020-657e-4b62-89cb-226f760442f5 2020-03-09T00:24:37.883 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/26980,v1:172.21.15.139:6801/26980] [v2:172.21.15.139:6802/26980,v1:172.21.15.139:6803/26980] exists,up 4103b4d7-7a38-48e4-8403-3549f5b3fda2 2020-03-09T00:24:37.883 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6802/28012,v1:172.21.15.104:6803/28012] [v2:172.21.15.104:6804/28012,v1:172.21.15.104:6805/28012] exists,up ce80df27-1b2e-41e1-9ded-a012ad488591 2020-03-09T00:24:37.883 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6800/27365,v1:172.21.15.148:6801/27365] [v2:172.21.15.148:6802/27365,v1:172.21.15.148:6803/27365] exists,up 8ff5a871-6f10-4e55-ac60-33bdc11c0292 2020-03-09T00:24:37.884 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6800/27630,v1:172.21.15.50:6801/27630] [v2:172.21.15.50:6802/27630,v1:172.21.15.50:6803/27630] exists,up 5122cc94-7217-4d84-8b81-11383b48c416 2020-03-09T00:24:37.884 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6809/26979,v1:172.21.15.139:6811/26979] [v2:172.21.15.139:6813/26979,v1:172.21.15.139:6815/26979] exists,up 1bea4f99-5e2a-4fbc-bac3-8cb45d678742 2020-03-09T00:24:37.884 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6818/28042,v1:172.21.15.104:6819/28042] [v2:172.21.15.104:6820/28042,v1:172.21.15.104:6822/28042] exists,up f6a853dd-1462-4f8c-83d6-701258d78519 2020-03-09T00:24:37.884 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6808/27364,v1:172.21.15.148:6809/27364] [v2:172.21.15.148:6810/27364,v1:172.21.15.148:6811/27364] exists,up b9911260-c1d0-4563-892b-35a9e1c47a9b 2020-03-09T00:24:37.884 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6821/27643,v1:172.21.15.50:6823/27643] [v2:172.21.15.50:6825/27643,v1:172.21.15.50:6827/27643] exists,up d7ebf4f6-a2b8-4076-83dd-504172481e32 2020-03-09T00:24:37.884 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10 up in weight 1 up_from 23 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/26978,v1:172.21.15.139:6810/26978] [v2:172.21.15.139:6812/26978,v1:172.21.15.139:6814/26978] exists,up cb76df3e-6ea9-4ee6-8b03-84e3cd329c69 2020-03-09T00:24:37.885 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6810/28013,v1:172.21.15.104:6811/28013] [v2:172.21.15.104:6812/28013,v1:172.21.15.104:6813/28013] exists,up 8bda27ba-cf17-472c-9a21-02dcaefab68f 2020-03-09T00:24:37.885 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6823/27395,v1:172.21.15.148:6825/27395] [v2:172.21.15.148:6826/27395,v1:172.21.15.148:6827/27395] exists,up d861da04-0be8-434b-9ed8-c1ec97fca7bd 2020-03-09T00:24:37.885 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6816/27631,v1:172.21.15.50:6817/27631] [v2:172.21.15.50:6818/27631,v1:172.21.15.50:6819/27631] exists,up 4ba3410e-df5b-4529-9ebe-34c15f68bc9d 2020-03-09T00:24:37.885 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6818/26981,v1:172.21.15.139:6821/26981] [v2:172.21.15.139:6824/26981,v1:172.21.15.139:6827/26981] exists,up 1ba69402-6d5c-491e-ac86-747517533a8b 2020-03-09T00:24:37.885 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] [v2:172.21.15.104:6825/28043,v1:172.21.15.104:6828/28043] exists,up a3566380-0c4a-403f-8533-b2d0dcfa8506 2020-03-09T00:24:37.893 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6816/27385,v1:172.21.15.148:6817/27385] [v2:172.21.15.148:6818/27385,v1:172.21.15.148:6819/27385] exists,up d0be76d1-3ce6-4c23-9917-41b35482def3', 'osd.1 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6808/27629,v1:172.21.15.50:6809/27629] [v2:172.21.15.50:6810/27629,v1:172.21.15.50:6811/27629] exists,up cee70020-657e-4b62-89cb-226f760442f5', 'osd.2 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/26980,v1:172.21.15.139:6801/26980] [v2:172.21.15.139:6802/26980,v1:172.21.15.139:6803/26980] exists,up 4103b4d7-7a38-48e4-8403-3549f5b3fda2', 'osd.3 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6802/28012,v1:172.21.15.104:6803/28012] [v2:172.21.15.104:6804/28012,v1:172.21.15.104:6805/28012] exists,up ce80df27-1b2e-41e1-9ded-a012ad488591', 'osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6800/27365,v1:172.21.15.148:6801/27365] [v2:172.21.15.148:6802/27365,v1:172.21.15.148:6803/27365] exists,up 8ff5a871-6f10-4e55-ac60-33bdc11c0292', 'osd.5 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6800/27630,v1:172.21.15.50:6801/27630] [v2:172.21.15.50:6802/27630,v1:172.21.15.50:6803/27630] exists,up 5122cc94-7217-4d84-8b81-11383b48c416', 'osd.6 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6809/26979,v1:172.21.15.139:6811/26979] [v2:172.21.15.139:6813/26979,v1:172.21.15.139:6815/26979] exists,up 1bea4f99-5e2a-4fbc-bac3-8cb45d678742', 'osd.7 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6818/28042,v1:172.21.15.104:6819/28042] [v2:172.21.15.104:6820/28042,v1:172.21.15.104:6822/28042] exists,up f6a853dd-1462-4f8c-83d6-701258d78519', 'osd.8 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6808/27364,v1:172.21.15.148:6809/27364] [v2:172.21.15.148:6810/27364,v1:172.21.15.148:6811/27364] exists,up b9911260-c1d0-4563-892b-35a9e1c47a9b', 'osd.9 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6821/27643,v1:172.21.15.50:6823/27643] [v2:172.21.15.50:6825/27643,v1:172.21.15.50:6827/27643] exists,up d7ebf4f6-a2b8-4076-83dd-504172481e32', 'osd.10 up in weight 1 up_from 23 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/26978,v1:172.21.15.139:6810/26978] [v2:172.21.15.139:6812/26978,v1:172.21.15.139:6814/26978] exists,up cb76df3e-6ea9-4ee6-8b03-84e3cd329c69', 'osd.11 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6810/28013,v1:172.21.15.104:6811/28013] [v2:172.21.15.104:6812/28013,v1:172.21.15.104:6813/28013] exists,up 8bda27ba-cf17-472c-9a21-02dcaefab68f', 'osd.12 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6823/27395,v1:172.21.15.148:6825/27395] [v2:172.21.15.148:6826/27395,v1:172.21.15.148:6827/27395] exists,up d861da04-0be8-434b-9ed8-c1ec97fca7bd', 'osd.13 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6816/27631,v1:172.21.15.50:6817/27631] [v2:172.21.15.50:6818/27631,v1:172.21.15.50:6819/27631] exists,up 4ba3410e-df5b-4529-9ebe-34c15f68bc9d', 'osd.14 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6818/26981,v1:172.21.15.139:6821/26981] [v2:172.21.15.139:6824/26981,v1:172.21.15.139:6827/26981] exists,up 1ba69402-6d5c-491e-ac86-747517533a8b', 'osd.15 up in weight 1 up_from 24 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] [v2:172.21.15.104:6825/28043,v1:172.21.15.104:6828/28043] exists,up a3566380-0c4a-403f-8533-b2d0dcfa8506'] 2020-03-09T00:24:37.894 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2020-03-09T00:24:38.003 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:38.003 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "mon.a", 2020-03-09T00:24:38.003 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster": "ceph", 2020-03-09T00:24:38.004 INFO:teuthology.orchestra.run.smithi148.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2020-03-09T00:24:38.004 INFO:teuthology.orchestra.run.smithi148.stdout: "admin_socket_mode": "", 2020-03-09T00:24:38.004 INFO:teuthology.orchestra.run.smithi148.stdout: "allow_ansi": "Terminal", 2020-03-09T00:24:38.004 INFO:teuthology.orchestra.run.smithi148.stdout: "auth_client_required": "cephx, none", 2020-03-09T00:24:38.004 INFO:teuthology.orchestra.run.smithi148.stdout: "auth_cluster_required": "cephx", 2020-03-09T00:24:38.004 INFO:teuthology.orchestra.run.smithi148.stdout: "auth_debug": "true", 2020-03-09T00:24:38.004 INFO:teuthology.orchestra.run.smithi148.stdout: "auth_mon_ticket_ttl": "43200.000000", 2020-03-09T00:24:38.005 INFO:teuthology.orchestra.run.smithi148.stdout: "auth_service_required": "cephx", 2020-03-09T00:24:38.005 INFO:teuthology.orchestra.run.smithi148.stdout: "auth_service_ticket_ttl": "3600.000000", 2020-03-09T00:24:38.005 INFO:teuthology.orchestra.run.smithi148.stdout: "auth_supported": "cephx", 2020-03-09T00:24:38.005 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_aio": "true", 2020-03-09T00:24:38.005 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_aio_max_queue_depth": "1024", 2020-03-09T00:24:38.005 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_aio_poll_ms": "250", 2020-03-09T00:24:38.005 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_aio_reap_max": "16", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_async_discard": "false", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_block_size": "4096", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_debug_aio": "false", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_debug_aio_log_age": "5.000000", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_debug_inflight_ios": "false", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_enable_discard": "false", 2020-03-09T00:24:38.006 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_inject_crash": "0", 2020-03-09T00:24:38.007 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_inject_crash_flush_delay": "2", 2020-03-09T00:24:38.007 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_nvme_retry_count": "-1", 2020-03-09T00:24:38.007 INFO:teuthology.orchestra.run.smithi148.stdout: "bdev_nvme_unbind_from_kernel": "false", 2020-03-09T00:24:38.007 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_alloc_size": "1048576", 2020-03-09T00:24:38.007 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_allocator": "bitmap", 2020-03-09T00:24:38.007 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_buffered_io": "true", 2020-03-09T00:24:38.007 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_compact_log_sync": "false", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_log_compact_min_size": "16777216", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_log_replay_check_allocations": "true", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_max_log_runway": "4194304", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_max_prefetch": "1048576", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_min_flush_size": "524288", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_min_log_runway": "1048576", 2020-03-09T00:24:38.008 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_preextend_wal_files": "true", 2020-03-09T00:24:38.009 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_shared_alloc_size": "65536", 2020-03-09T00:24:38.009 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_sync_write": "false", 2020-03-09T00:24:38.009 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2020-03-09T00:24:38.009 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2020-03-09T00:24:38.009 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2020-03-09T00:24:38.009 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_allocator": "bitmap", 2020-03-09T00:24:38.010 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2020-03-09T00:24:38.010 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2020-03-09T00:24:38.010 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2020-03-09T00:24:38.010 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bitmapallocator_span_size": "1024", 2020-03-09T00:24:38.010 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blobid_prealloc": "10240", 2020-03-09T00:24:38.010 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_create": "true", 2020-03-09T00:24:38.010 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_db_create": "false", 2020-03-09T00:24:38.011 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_db_path": "", 2020-03-09T00:24:38.011 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_db_size": "0", 2020-03-09T00:24:38.011 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_path": "", 2020-03-09T00:24:38.011 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_preallocate_file": "false", 2020-03-09T00:24:38.011 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_size": "107374182400", 2020-03-09T00:24:38.012 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_wal_create": "false", 2020-03-09T00:24:38.012 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_wal_path": "", 2020-03-09T00:24:38.012 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_block_wal_size": "100663296", 2020-03-09T00:24:38.012 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs": "true", 2020-03-09T00:24:38.012 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2020-03-09T00:24:38.012 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_balance_interval": "1.000000", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_db_compatibility": "true", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_env_mirror": "false", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_gift_ratio": "0.020000", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_max_free": "10737418240", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_max_ratio": "0.900000", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_min": "1073741824", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_min_free": "1073741824", 2020-03-09T00:24:38.013 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_min_ratio": "0.020000", 2020-03-09T00:24:38.014 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_bluefs_reclaim_ratio": "0.200000", 2020-03-09T00:24:38.014 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_autotune": "true", 2020-03-09T00:24:38.014 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_autotune_interval": "5.000000", 2020-03-09T00:24:38.014 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_kv_ratio": "0.400000", 2020-03-09T00:24:38.014 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_ratio": "0.400000", 2020-03-09T00:24:38.014 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_size": "0", 2020-03-09T00:24:38.014 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_size_hdd": "1073741824", 2020-03-09T00:24:38.015 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_size_ssd": "3221225472", 2020-03-09T00:24:38.015 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_trim_interval": "0.050000", 2020-03-09T00:24:38.015 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_trim_max_skip_pinned": "64", 2020-03-09T00:24:38.015 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_type": "2q", 2020-03-09T00:24:38.015 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_clone_cow": "true", 2020-03-09T00:24:38.015 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_algorithm": "snappy", 2020-03-09T00:24:38.015 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_max_blob_size": "0", 2020-03-09T00:24:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_max_blob_size_hdd": "524288", 2020-03-09T00:24:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2020-03-09T00:24:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_min_blob_size": "0", 2020-03-09T00:24:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_min_blob_size_hdd": "131072", 2020-03-09T00:24:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2020-03-09T00:24:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_mode": "none", 2020-03-09T00:24:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_compression_required_ratio": "0.875000", 2020-03-09T00:24:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_csum_type": "crc32c", 2020-03-09T00:24:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_enforce_settings": "default", 2020-03-09T00:24:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_freelist": "false", 2020-03-09T00:24:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_fsck_abort": "false", 2020-03-09T00:24:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_inject_bug21040": "false", 2020-03-09T00:24:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2020-03-09T00:24:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_inject_read_err": "false", 2020-03-09T00:24:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_misc": "false", 2020-03-09T00:24:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_no_reuse_blocks": "false", 2020-03-09T00:24:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_omit_block_device_write": "false", 2020-03-09T00:24:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_omit_kv_commit": "false", 2020-03-09T00:24:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2020-03-09T00:24:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_prefill": "0.000000", 2020-03-09T00:24:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_prefragment_max": "1048576", 2020-03-09T00:24:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_random_read_err": "0.000000", 2020-03-09T00:24:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2020-03-09T00:24:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_small_allocations": "0", 2020-03-09T00:24:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2020-03-09T00:24:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_default_buffered_read": "true", 2020-03-09T00:24:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_default_buffered_write": "false", 2020-03-09T00:24:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_deferred_batch_ops": "0", 2020-03-09T00:24:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2020-03-09T00:24:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2020-03-09T00:24:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2020-03-09T00:24:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_map_shard_max_size": "1200", 2020-03-09T00:24:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_map_shard_min_size": "150", 2020-03-09T00:24:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_map_shard_target_size": "500", 2020-03-09T00:24:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2020-03-09T00:24:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_freelist_blocks_per_key": "128", 2020-03-09T00:24:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2020-03-09T00:24:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2020-03-09T00:24:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_on_mkfs": "true", 2020-03-09T00:24:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2020-03-09T00:24:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_on_mount": "false", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_on_mount_deep": "false", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_on_umount": "false", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_on_umount_deep": "false", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_quick_fix_on_mount": "true", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_quick_fix_threads": "2", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_gc_enable_blob_threshold": "0", 2020-03-09T00:24:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_gc_enable_total_threshold": "0", 2020-03-09T00:24:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_ignore_data_csum": "false", 2020-03-09T00:24:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_ioring": "false", 2020-03-09T00:24:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_kvbackend": "rocksdb", 2020-03-09T00:24:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_log_collection_list_age": "60.000000", 2020-03-09T00:24:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2020-03-09T00:24:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_log_op_age": "5.000000", 2020-03-09T00:24:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_max_alloc_size": "0", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_max_blob_size": "0", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_max_blob_size_hdd": "524288", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_max_blob_size_ssd": "65536", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_max_defer_interval": "3.000000", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_max_deferred_txc": "32", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_min_alloc_size": "0", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_min_alloc_size_hdd": "65536", 2020-03-09T00:24:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_min_alloc_size_ssd": "4096", 2020-03-09T00:24:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_nid_prealloc": "1024", 2020-03-09T00:24:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_prefer_deferred_size": "0", 2020-03-09T00:24:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_prefer_deferred_size_hdd": "32768", 2020-03-09T00:24:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2020-03-09T00:24:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_retry_disk_reads": "3", 2020-03-09T00:24:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_rocksdb_cf": "false", 2020-03-09T00:24:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_rocksdb_cfs": "M= P= L=", 2020-03-09T00:24:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2", 2020-03-09T00:24:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_spdk_coremask": "0x1", 2020-03-09T00:24:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_spdk_io_sleep": "5", 2020-03-09T00:24:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_spdk_max_io_completion": "0", 2020-03-09T00:24:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_spdk_mem": "512", 2020-03-09T00:24:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_sync_submit_transaction": "false", 2020-03-09T00:24:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_throttle_bytes": "67108864", 2020-03-09T00:24:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_throttle_cost_per_io": "0", 2020-03-09T00:24:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2020-03-09T00:24:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2020-03-09T00:24:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2020-03-09T00:24:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_throttle_trace_rate": "0.000000", 2020-03-09T00:24:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_tracing": "false", 2020-03-09T00:24:38.028 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_volume_selection_policy": "rocksdb_original", 2020-03-09T00:24:38.028 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_volume_selection_reserved": "0", 2020-03-09T00:24:38.028 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2020-03-09T00:24:38.028 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2020-03-09T00:24:38.028 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_warn_on_legacy_statfs": "true", 2020-03-09T00:24:38.028 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2020-03-09T00:24:38.028 INFO:teuthology.orchestra.run.smithi148.stdout: "cephadm_path": "/usr/sbin/cephadm", 2020-03-09T00:24:38.029 INFO:teuthology.orchestra.run.smithi148.stdout: "cephx_cluster_require_signatures": "false", 2020-03-09T00:24:38.029 INFO:teuthology.orchestra.run.smithi148.stdout: "cephx_cluster_require_version": "1", 2020-03-09T00:24:38.029 INFO:teuthology.orchestra.run.smithi148.stdout: "cephx_require_signatures": "false", 2020-03-09T00:24:38.029 INFO:teuthology.orchestra.run.smithi148.stdout: "cephx_require_version": "1", 2020-03-09T00:24:38.029 INFO:teuthology.orchestra.run.smithi148.stdout: "cephx_service_require_signatures": "false", 2020-03-09T00:24:38.029 INFO:teuthology.orchestra.run.smithi148.stdout: "cephx_service_require_version": "1", 2020-03-09T00:24:38.030 INFO:teuthology.orchestra.run.smithi148.stdout: "cephx_sign_messages": "true", 2020-03-09T00:24:38.030 INFO:teuthology.orchestra.run.smithi148.stdout: "chdir": "", 2020-03-09T00:24:38.030 INFO:teuthology.orchestra.run.smithi148.stdout: "client_acl_type": "", 2020-03-09T00:24:38.030 INFO:teuthology.orchestra.run.smithi148.stdout: "client_cache_mid": "0.750000", 2020-03-09T00:24:38.030 INFO:teuthology.orchestra.run.smithi148.stdout: "client_cache_size": "16384", 2020-03-09T00:24:38.030 INFO:teuthology.orchestra.run.smithi148.stdout: "client_caps_release_delay": "5", 2020-03-09T00:24:38.030 INFO:teuthology.orchestra.run.smithi148.stdout: "client_check_pool_perm": "true", 2020-03-09T00:24:38.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client_debug_force_sync_read": "false", 2020-03-09T00:24:38.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client_debug_getattr_caps": "false", 2020-03-09T00:24:38.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client_debug_inject_tick_delay": "0", 2020-03-09T00:24:38.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client_die_on_failed_dentry_invalidate": "true", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_die_on_failed_remount": "false", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_dirsize_rbytes": "true", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_force_lazyio": "false", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_fs": "", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_inject_fixed_oldest_tid": "false", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_inject_release_failure": "false", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_max_inline_size": "4096", 2020-03-09T00:24:38.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_mds_namespace": "", 2020-03-09T00:24:38.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_metadata": "", 2020-03-09T00:24:38.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_mount_gid": "-1", 2020-03-09T00:24:38.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_mount_timeout": "300.000000", 2020-03-09T00:24:38.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_mount_uid": "-1", 2020-03-09T00:24:38.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_mountpoint": "/", 2020-03-09T00:24:38.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_notify_timeout": "10", 2020-03-09T00:24:38.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_oc": "true", 2020-03-09T00:24:38.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_oc_max_dirty": "104857600", 2020-03-09T00:24:38.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_oc_max_dirty_age": "5.000000", 2020-03-09T00:24:38.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_oc_max_objects": "1000", 2020-03-09T00:24:38.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_oc_size": "209715200", 2020-03-09T00:24:38.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_oc_target_dirty": "8388608", 2020-03-09T00:24:38.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_permissions": "true", 2020-03-09T00:24:38.034 INFO:teuthology.orchestra.run.smithi148.stdout: "client_quota_df": "true", 2020-03-09T00:24:38.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_readahead_max_bytes": "0", 2020-03-09T00:24:38.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_readahead_max_periods": "4", 2020-03-09T00:24:38.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_readahead_min": "131072", 2020-03-09T00:24:38.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_reconnect_stale": "false", 2020-03-09T00:24:38.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_snapdir": ".snap", 2020-03-09T00:24:38.035 INFO:teuthology.orchestra.run.smithi148.stdout: "client_tick_interval": "1.000000", 2020-03-09T00:24:38.036 INFO:teuthology.orchestra.run.smithi148.stdout: "client_trace": "", 2020-03-09T00:24:38.036 INFO:teuthology.orchestra.run.smithi148.stdout: "client_try_dentry_invalidate": "false", 2020-03-09T00:24:38.036 INFO:teuthology.orchestra.run.smithi148.stdout: "client_use_faked_inos": "false", 2020-03-09T00:24:38.036 INFO:teuthology.orchestra.run.smithi148.stdout: "client_use_random_mds": "false", 2020-03-09T00:24:38.036 INFO:teuthology.orchestra.run.smithi148.stdout: "clog_to_graylog": "false", 2020-03-09T00:24:38.036 INFO:teuthology.orchestra.run.smithi148.stdout: "clog_to_graylog_host": "127.0.0.1", 2020-03-09T00:24:38.036 INFO:teuthology.orchestra.run.smithi148.stdout: "clog_to_graylog_port": "12201", 2020-03-09T00:24:38.037 INFO:teuthology.orchestra.run.smithi148.stdout: "clog_to_monitors": "default=true", 2020-03-09T00:24:38.037 INFO:teuthology.orchestra.run.smithi148.stdout: "clog_to_syslog": "false", 2020-03-09T00:24:38.037 INFO:teuthology.orchestra.run.smithi148.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2020-03-09T00:24:38.037 INFO:teuthology.orchestra.run.smithi148.stdout: "clog_to_syslog_level": "info", 2020-03-09T00:24:38.037 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "-", 2020-03-09T00:24:38.037 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_network": "", 2020-03-09T00:24:38.037 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_network_interface": "", 2020-03-09T00:24:38.038 INFO:teuthology.orchestra.run.smithi148.stdout: "colors": "Terminal", 2020-03-09T00:24:38.038 INFO:teuthology.orchestra.run.smithi148.stdout: "compressor_zlib_isal": "false", 2020-03-09T00:24:38.038 INFO:teuthology.orchestra.run.smithi148.stdout: "compressor_zlib_level": "5", 2020-03-09T00:24:38.038 INFO:teuthology.orchestra.run.smithi148.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2020-03-09T00:24:38.038 INFO:teuthology.orchestra.run.smithi148.stdout: "continuation_prompt": ">", 2020-03-09T00:24:38.038 INFO:teuthology.orchestra.run.smithi148.stdout: "crash_dir": "/var/lib/ceph/crash", 2020-03-09T00:24:38.038 INFO:teuthology.orchestra.run.smithi148.stdout: "crimson_osd_obc_lru_size": "10", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location_hook": "", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location_hook_timeout": "10", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "daemonize": "false", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_allow_any_pool_priority": "false", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_asok": "1/5", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_asok_assert_abort": "false", 2020-03-09T00:24:38.039 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_asserts_on_shutdown": "true", 2020-03-09T00:24:38.040 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_auth": "20/20", 2020-03-09T00:24:38.040 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_bdev": "1/3", 2020-03-09T00:24:38.040 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_bluefs": "1/5", 2020-03-09T00:24:38.040 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_bluestore": "1/5", 2020-03-09T00:24:38.040 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_buffer": "0/1", 2020-03-09T00:24:38.040 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_civetweb": "1/10", 2020-03-09T00:24:38.040 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_client": "0/5", 2020-03-09T00:24:38.041 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_compressor": "1/5", 2020-03-09T00:24:38.041 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_context": "0/1", 2020-03-09T00:24:38.041 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_crush": "1/1", 2020-03-09T00:24:38.041 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_crypto": "1/5", 2020-03-09T00:24:38.041 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_deliberately_leak_memory": "false", 2020-03-09T00:24:38.041 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_disable_randomized_ping": "false", 2020-03-09T00:24:38.042 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_dpdk": "1/5", 2020-03-09T00:24:38.042 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_eventtrace": "1/5", 2020-03-09T00:24:38.042 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_filer": "0/1", 2020-03-09T00:24:38.042 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_filestore": "1/3", 2020-03-09T00:24:38.042 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_finisher": "1/1", 2020-03-09T00:24:38.042 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_fuse": "1/5", 2020-03-09T00:24:38.042 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_heartbeat_testing_span": "0", 2020-03-09T00:24:38.043 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_heartbeatmap": "1/5", 2020-03-09T00:24:38.043 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_immutable_obj_cache": "0/5", 2020-03-09T00:24:38.043 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_javaclient": "1/5", 2020-03-09T00:24:38.043 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_journal": "1/3", 2020-03-09T00:24:38.043 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_journaler": "0/5", 2020-03-09T00:24:38.043 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_kstore": "1/5", 2020-03-09T00:24:38.044 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_leveldb": "4/5", 2020-03-09T00:24:38.044 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_lockdep": "0/1", 2020-03-09T00:24:38.044 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mds": "1/5", 2020-03-09T00:24:38.044 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mds_balancer": "1/5", 2020-03-09T00:24:38.044 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mds_locker": "1/5", 2020-03-09T00:24:38.044 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mds_log": "1/5", 2020-03-09T00:24:38.045 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mds_log_expire": "1/5", 2020-03-09T00:24:38.045 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mds_migrator": "1/5", 2020-03-09T00:24:38.045 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_memdb": "4/5", 2020-03-09T00:24:38.045 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mgr": "1/5", 2020-03-09T00:24:38.045 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mgrc": "1/5", 2020-03-09T00:24:38.045 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_mon": "20/20", 2020-03-09T00:24:38.045 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_monc": "0/10", 2020-03-09T00:24:38.046 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_ms": "1/1", 2020-03-09T00:24:38.046 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_none": "0/5", 2020-03-09T00:24:38.046 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_objclass": "0/5", 2020-03-09T00:24:38.046 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_objectcacher": "0/5", 2020-03-09T00:24:38.046 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_objecter": "0/1", 2020-03-09T00:24:38.046 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_optracker": "0/5", 2020-03-09T00:24:38.046 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_osd": "1/5", 2020-03-09T00:24:38.047 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_paxos": "20/20", 2020-03-09T00:24:38.047 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_perfcounter": "1/5", 2020-03-09T00:24:38.047 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_prioritycache": "1/5", 2020-03-09T00:24:38.047 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rados": "0/5", 2020-03-09T00:24:38.047 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rbd": "0/5", 2020-03-09T00:24:38.047 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rbd_mirror": "0/5", 2020-03-09T00:24:38.047 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rbd_replay": "0/5", 2020-03-09T00:24:38.048 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rbd_rwl": "0/5", 2020-03-09T00:24:38.048 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_refs": "0/0", 2020-03-09T00:24:38.048 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_reserver": "1/1", 2020-03-09T00:24:38.048 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rgw": "1/5", 2020-03-09T00:24:38.048 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rgw_sync": "1/5", 2020-03-09T00:24:38.048 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_rocksdb": "4/5", 2020-03-09T00:24:38.048 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_shell": "false", 2020-03-09T00:24:38.049 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_striper": "0/1", 2020-03-09T00:24:38.049 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_test": "0/5", 2020-03-09T00:24:38.049 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_throttle": "1/1", 2020-03-09T00:24:38.049 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_timer": "0/1", 2020-03-09T00:24:38.049 INFO:teuthology.orchestra.run.smithi148.stdout: "debug_tp": "0/5", 2020-03-09T00:24:38.049 INFO:teuthology.orchestra.run.smithi148.stdout: "device_failure_prediction_mode": "none", 2020-03-09T00:24:38.049 INFO:teuthology.orchestra.run.smithi148.stdout: "echo": "false", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "editor": "vim", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "err_to_graylog": "false", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "err_to_stderr": "true", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "err_to_syslog": "false", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event_tracing": "false", 2020-03-09T00:24:38.050 INFO:teuthology.orchestra.run.smithi148.stdout: "fake_statfs_for_testing": "0", 2020-03-09T00:24:38.051 INFO:teuthology.orchestra.run.smithi148.stdout: "fatal_signal_handlers": "true", 2020-03-09T00:24:38.051 INFO:teuthology.orchestra.run.smithi148.stdout: "feedback_to_output": "false", 2020-03-09T00:24:38.051 INFO:teuthology.orchestra.run.smithi148.stdout: "filer_max_purge_ops": "10", 2020-03-09T00:24:38.051 INFO:teuthology.orchestra.run.smithi148.stdout: "filer_max_truncate_ops": "128", 2020-03-09T00:24:38.051 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_apply_finisher_threads": "1", 2020-03-09T00:24:38.051 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_blackhole": "false", 2020-03-09T00:24:38.051 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_btrfs_clone_range": "true", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_btrfs_snap": "true", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_caller_concurrency": "10", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_collect_device_partition_information": "true", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_commit_timeout": "600.000000", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_debug_inject_read_err": "false", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_debug_omap_check": "false", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_debug_random_read_err": "0.000000", 2020-03-09T00:24:38.052 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_debug_verify_split": "false", 2020-03-09T00:24:38.053 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_dump_file": "", 2020-03-09T00:24:38.053 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2020-03-09T00:24:38.053 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_expected_throughput_ops": "200.000000", 2020-03-09T00:24:38.053 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_fadvise": "true", 2020-03-09T00:24:38.053 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_fail_eio": "true", 2020-03-09T00:24:38.053 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_fd_cache_shards": "16", 2020-03-09T00:24:38.053 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_fd_cache_size": "128", 2020-03-09T00:24:38.054 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_fiemap": "false", 2020-03-09T00:24:38.054 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_fiemap_threshold": "4096", 2020-03-09T00:24:38.054 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_fsync_flushes_journal_data": "false", 2020-03-09T00:24:38.054 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_index_retry_probability": "0.000000", 2020-03-09T00:24:38.054 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_inject_stall": "0", 2020-03-09T00:24:38.054 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_journal_parallel": "false", 2020-03-09T00:24:38.054 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_journal_trailing": "false", 2020-03-09T00:24:38.055 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_journal_writeahead": "false", 2020-03-09T00:24:38.055 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_kill_at": "0", 2020-03-09T00:24:38.055 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_alloc_hint_size": "1048576", 2020-03-09T00:24:38.055 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattr_size": "0", 2020-03-09T00:24:38.055 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2020-03-09T00:24:38.056 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattr_size_other": "512", 2020-03-09T00:24:38.056 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2020-03-09T00:24:38.056 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattrs": "0", 2020-03-09T00:24:38.056 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2020-03-09T00:24:38.056 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattrs_other": "2", 2020-03-09T00:24:38.057 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_inline_xattrs_xfs": "10", 2020-03-09T00:24:38.057 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_sync_interval": "5.000000", 2020-03-09T00:24:38.057 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_xattr_value_size": "0", 2020-03-09T00:24:38.057 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2020-03-09T00:24:38.057 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_xattr_value_size_other": "1024", 2020-03-09T00:24:38.057 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2020-03-09T00:24:38.057 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_merge_threshold": "-10", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_min_sync_interval": "0.010000", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_odsync_write": "false", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_omap_backend": "rocksdb", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_omap_backend_path": "", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_omap_header_cache_size": "1024", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_ondisk_finisher_threads": "1", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_op_thread_suicide_timeout": "180", 2020-03-09T00:24:38.058 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_op_thread_timeout": "60", 2020-03-09T00:24:38.059 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_op_threads": "2", 2020-03-09T00:24:38.059 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_punch_hole": "false", 2020-03-09T00:24:38.059 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2020-03-09T00:24:38.059 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2020-03-09T00:24:38.059 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2020-03-09T00:24:38.059 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_high_threshhold": "0.900000", 2020-03-09T00:24:38.059 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_low_threshhold": "0.300000", 2020-03-09T00:24:38.060 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_max_bytes": "104857600", 2020-03-09T00:24:38.060 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2020-03-09T00:24:38.060 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2020-03-09T00:24:38.060 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2020-03-09T00:24:38.060 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_queue_max_ops": "50", 2020-03-09T00:24:38.060 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2020-03-09T00:24:38.061 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_seek_data_hole": "false", 2020-03-09T00:24:38.061 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_sloppy_crc": "false", 2020-03-09T00:24:38.061 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_sloppy_crc_block_size": "65536", 2020-03-09T00:24:38.061 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_splice": "false", 2020-03-09T00:24:38.061 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_split_multiple": "2", 2020-03-09T00:24:38.061 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_split_rand_factor": "20", 2020-03-09T00:24:38.061 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_update_to": "1000", 2020-03-09T00:24:38.062 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2020-03-09T00:24:38.062 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2020-03-09T00:24:38.062 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2020-03-09T00:24:38.062 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2020-03-09T00:24:38.062 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2020-03-09T00:24:38.062 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2020-03-09T00:24:38.063 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_enable": "true", 2020-03-09T00:24:38.063 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2020-03-09T00:24:38.063 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2020-03-09T00:24:38.063 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2020-03-09T00:24:38.063 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2020-03-09T00:24:38.063 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2020-03-09T00:24:38.063 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2020-03-09T00:24:38.064 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_xfs_extsize": "false", 2020-03-09T00:24:38.064 INFO:teuthology.orchestra.run.smithi148.stdout: "filestore_zfs_snap": "false", 2020-03-09T00:24:38.064 INFO:teuthology.orchestra.run.smithi148.stdout: "fio_dir": "/tmp/fio", 2020-03-09T00:24:38.064 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "036b76b6-511e-4ec6-9669-0a47550bd261", 2020-03-09T00:24:38.064 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_allow_other": "true", 2020-03-09T00:24:38.065 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_atomic_o_trunc": "true", 2020-03-09T00:24:38.065 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_big_writes": "false", 2020-03-09T00:24:38.065 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_debug": "false", 2020-03-09T00:24:38.065 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_default_permissions": "false", 2020-03-09T00:24:38.066 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_disable_pagecache": "false", 2020-03-09T00:24:38.066 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_max_write": "0", 2020-03-09T00:24:38.066 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_multithreaded": "true", 2020-03-09T00:24:38.066 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_require_active_mds": "true", 2020-03-09T00:24:38.066 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_set_user_groups": "true", 2020-03-09T00:24:38.066 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_syncfs_on_mksnap": "true", 2020-03-09T00:24:38.066 INFO:teuthology.orchestra.run.smithi148.stdout: "fuse_use_invalidate_cb": "true", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "gss_target_name": "ceph", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_file": "", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_inject_failure": "0", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_interval": "5", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "host": "", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2020-03-09T00:24:38.067 INFO:teuthology.orchestra.run.smithi148.stdout: "immutable_object_cache_max_inflight_ops": "128", 2020-03-09T00:24:38.068 INFO:teuthology.orchestra.run.smithi148.stdout: "immutable_object_cache_max_size": "1073741824", 2020-03-09T00:24:38.068 INFO:teuthology.orchestra.run.smithi148.stdout: "immutable_object_cache_path": "/tmp", 2020-03-09T00:24:38.068 INFO:teuthology.orchestra.run.smithi148.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2020-03-09T00:24:38.068 INFO:teuthology.orchestra.run.smithi148.stdout: "immutable_object_cache_watermark": "0.100000", 2020-03-09T00:24:38.068 INFO:teuthology.orchestra.run.smithi148.stdout: "inject_early_sigterm": "false", 2020-03-09T00:24:38.068 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_aio": "true", 2020-03-09T00:24:38.068 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_align_min_size": "65536", 2020-03-09T00:24:38.069 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_block_align": "true", 2020-03-09T00:24:38.069 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_block_size": "4096", 2020-03-09T00:24:38.069 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_dio": "true", 2020-03-09T00:24:38.069 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_discard": "false", 2020-03-09T00:24:38.069 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_force_aio": "false", 2020-03-09T00:24:38.069 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_ignore_corruption": "false", 2020-03-09T00:24:38.069 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_max_write_bytes": "10485760", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_max_write_entries": "100", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_replay_from": "0", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_throttle_high_multiple": "0.000000", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_throttle_high_threshhold": "0.900000", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_throttle_low_threshhold": "0.600000", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_throttle_max_multiple": "0.000000", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_write_header_frequency": "0", 2020-03-09T00:24:38.070 INFO:teuthology.orchestra.run.smithi148.stdout: "journal_zero_on_create": "false", 2020-03-09T00:24:38.071 INFO:teuthology.orchestra.run.smithi148.stdout: "journaler_prefetch_periods": "10", 2020-03-09T00:24:38.071 INFO:teuthology.orchestra.run.smithi148.stdout: "journaler_prezero_periods": "5", 2020-03-09T00:24:38.071 INFO:teuthology.orchestra.run.smithi148.stdout: "journaler_write_head_interval": "15", 2020-03-09T00:24:38.071 INFO:teuthology.orchestra.run.smithi148.stdout: "key": "", 2020-03-09T00:24:38.071 INFO:teuthology.orchestra.run.smithi148.stdout: "keyfile": "", 2020-03-09T00:24:38.071 INFO:teuthology.orchestra.run.smithi148.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2020-03-09T00:24:38.071 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_backend": "rocksdb", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_default_stripe_size": "65536", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_fsck_on_mount": "false", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_fsck_on_mount_deep": "true", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_max_bytes": "67108864", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_max_ops": "512", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_nid_prealloc": "1024", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_onode_map_size": "1024", 2020-03-09T00:24:38.072 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2020-03-09T00:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_sync_submit_transaction": "false", 2020-03-09T00:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout: "kstore_sync_transaction": "false", 2020-03-09T00:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_block_size": "65536", 2020-03-09T00:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_bloom_size": "0", 2020-03-09T00:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_cache_size": "536870912", 2020-03-09T00:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_compact_on_mount": "false", 2020-03-09T00:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_compression": "false", 2020-03-09T00:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_log": "", 2020-03-09T00:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_log_to_ceph_log": "true", 2020-03-09T00:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_max_open_files": "0", 2020-03-09T00:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_paranoid": "false", 2020-03-09T00:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout: "leveldb_write_buffer_size": "33554432", 2020-03-09T00:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout: "lockdep": "false", 2020-03-09T00:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout: "lockdep_force_backtrace": "false", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_coarse_timestamps": "true", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_flush_on_exit": "false", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_graylog_host": "127.0.0.1", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_graylog_port": "12201", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_max_new": "1000", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_max_recent": "10000", 2020-03-09T00:24:38.075 INFO:teuthology.orchestra.run.smithi148.stdout: "log_stderr_prefix": "", 2020-03-09T00:24:38.076 INFO:teuthology.orchestra.run.smithi148.stdout: "log_stop_at_utilization": "0.970000", 2020-03-09T00:24:38.076 INFO:teuthology.orchestra.run.smithi148.stdout: "log_to_file": "true", 2020-03-09T00:24:38.076 INFO:teuthology.orchestra.run.smithi148.stdout: "log_to_graylog": "false", 2020-03-09T00:24:38.076 INFO:teuthology.orchestra.run.smithi148.stdout: "log_to_stderr": "false", 2020-03-09T00:24:38.076 INFO:teuthology.orchestra.run.smithi148.stdout: "log_to_syslog": "false", 2020-03-09T00:24:38.076 INFO:teuthology.orchestra.run.smithi148.stdout: "max_completion_items": "50", 2020-03-09T00:24:38.077 INFO:teuthology.orchestra.run.smithi148.stdout: "max_rotating_auth_attempts": "10", 2020-03-09T00:24:38.077 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_action_on_write_error": "1", 2020-03-09T00:24:38.077 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_export_pin": "true", 2020-03-09T00:24:38.077 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_fragment_dirs": "true", 2020-03-09T00:24:38.077 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2020-03-09T00:24:38.077 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_fragment_interval": "5", 2020-03-09T00:24:38.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_fragment_size_max": "100000", 2020-03-09T00:24:38.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_idle_threshold": "0.000000", 2020-03-09T00:24:38.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_interval": "10", 2020-03-09T00:24:38.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_max": "-1", 2020-03-09T00:24:38.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_max_until": "-1", 2020-03-09T00:24:38.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_merge_size": "50", 2020-03-09T00:24:38.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_midchunk": "0.300000", 2020-03-09T00:24:38.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_min_rebalance": "0.100000", 2020-03-09T00:24:38.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_min_start": "0.200000", 2020-03-09T00:24:38.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_minchunk": "0.001000", 2020-03-09T00:24:38.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_mode": "0", 2020-03-09T00:24:38.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_need_max": "1.200000", 2020-03-09T00:24:38.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_need_min": "0.800000", 2020-03-09T00:24:38.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_replicate_threshold": "8000.000000", 2020-03-09T00:24:38.080 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_sample_interval": "3.000000", 2020-03-09T00:24:38.080 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_split_bits": "3", 2020-03-09T00:24:38.080 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_split_rd": "25000.000000", 2020-03-09T00:24:38.080 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_split_size": "10000", 2020-03-09T00:24:38.080 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_split_wr": "10000.000000", 2020-03-09T00:24:38.080 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_target_decay": "10.000000", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_beacon_grace": "15.000000", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_beacon_interval": "4.000000", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache_memory_limit": "4294967296", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache_mid": "0.700000", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache_release_free_interval": "10", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache_reservation": "0.050000", 2020-03-09T00:24:38.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache_trim_decay_rate": "1.000000", 2020-03-09T00:24:38.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache_trim_interval": "1", 2020-03-09T00:24:38.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache_trim_threshold": "65536", 2020-03-09T00:24:38.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2020-03-09T00:24:38.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_client_delegate_inos_pct": "50", 2020-03-09T00:24:38.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_client_prealloc_inos": "1000", 2020-03-09T00:24:38.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2020-03-09T00:24:38.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_damage_table_max_entries": "10000", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_debug_auth_pins": "false", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_debug_frag": "false", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_debug_scatterstat": "false", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_debug_subtrees": "false", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_decay_halflife": "5.000000", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_default_dir_hash": "2", 2020-03-09T00:24:38.083 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_defer_session_stale": "true", 2020-03-09T00:24:38.084 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_dir_keys_per_op": "16384", 2020-03-09T00:24:38.084 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_dir_max_commit_size": "10", 2020-03-09T00:24:38.084 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_dirstat_min_interval": "1.000000", 2020-03-09T00:24:38.084 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_dump_cache_after_rejoin": "false", 2020-03-09T00:24:38.084 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_dump_cache_on_map": "false", 2020-03-09T00:24:38.084 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_dump_cache_threshold_file": "0", 2020-03-09T00:24:38.084 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2020-03-09T00:24:38.085 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_early_reply": "true", 2020-03-09T00:24:38.085 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_enable_op_tracker": "true", 2020-03-09T00:24:38.085 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_enforce_unique_name": "true", 2020-03-09T00:24:38.085 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_forward_all_requests_to_auth": "false", 2020-03-09T00:24:38.085 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_freeze_tree_timeout": "30.000000", 2020-03-09T00:24:38.085 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2020-03-09T00:24:38.085 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_health_cache_threshold": "1.500000", 2020-03-09T00:24:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_health_summarize_threshold": "10", 2020-03-09T00:24:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_heartbeat_grace": "15.000000", 2020-03-09T00:24:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_inject_migrator_session_race": "false", 2020-03-09T00:24:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2020-03-09T00:24:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_join_fs": "", 2020-03-09T00:24:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_journal_format": "1", 2020-03-09T00:24:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_create_at": "0", 2020-03-09T00:24:38.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_export_at": "0", 2020-03-09T00:24:38.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_import_at": "0", 2020-03-09T00:24:38.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_journal_at": "0", 2020-03-09T00:24:38.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_journal_expire_at": "0", 2020-03-09T00:24:38.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_journal_replay_at": "0", 2020-03-09T00:24:38.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_link_at": "0", 2020-03-09T00:24:38.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_mdstable_at": "0", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_openc_at": "0", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_kill_rename_at": "0", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log_events_per_segment": "1024", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log_max_events": "-1", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log_max_segments": "128", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log_pause": "false", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log_segment_size": "0", 2020-03-09T00:24:38.088 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log_skip_corrupt_events": "false", 2020-03-09T00:24:38.089 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_caps_per_client": "1048576", 2020-03-09T00:24:38.089 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_completed_flushes": "100000", 2020-03-09T00:24:38.089 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_completed_requests": "100000", 2020-03-09T00:24:38.089 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_export_size": "20971520", 2020-03-09T00:24:38.089 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_file_recover": "32", 2020-03-09T00:24:38.089 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_purge_files": "64", 2020-03-09T00:24:38.089 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_purge_ops": "8192", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_retries_on_remount_failure": "5", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_scrub_ops_in_progress": "5", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_snaps_per_dir": "100", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_max_xattr_pairs_size": "65536", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_min_caps_per_client": "100", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_mon_shutdown_timeout": "5.000000", 2020-03-09T00:24:38.090 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_numa_node": "-1", 2020-03-09T00:24:38.091 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_op_complaint_time": "30.000000", 2020-03-09T00:24:38.091 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_op_history_duration": "600", 2020-03-09T00:24:38.091 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_op_history_size": "20", 2020-03-09T00:24:38.091 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_op_log_threshold": "5", 2020-03-09T00:24:38.091 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2020-03-09T00:24:38.091 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_recall_global_max_decay_threshold": "65536", 2020-03-09T00:24:38.091 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_recall_max_caps": "5000", 2020-03-09T00:24:38.092 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_recall_max_decay_rate": "2.500000", 2020-03-09T00:24:38.092 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_recall_max_decay_threshold": "16384", 2020-03-09T00:24:38.092 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_recall_warning_decay_rate": "60.000000", 2020-03-09T00:24:38.092 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_recall_warning_threshold": "32768", 2020-03-09T00:24:38.092 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_reconnect_timeout": "45.000000", 2020-03-09T00:24:38.092 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_replay_interval": "1.000000", 2020-03-09T00:24:38.092 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_replay_unsafe_with_closed_session": "false", 2020-03-09T00:24:38.093 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_request_load_average_decay_rate": "60.000000", 2020-03-09T00:24:38.093 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_root_ino_gid": "0", 2020-03-09T00:24:38.093 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_root_ino_uid": "0", 2020-03-09T00:24:38.093 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_scatter_nudge_interval": "5.000000", 2020-03-09T00:24:38.094 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_session_blacklist_on_evict": "true", 2020-03-09T00:24:38.094 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_session_blacklist_on_timeout": "true", 2020-03-09T00:24:38.094 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2020-03-09T00:24:38.094 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_session_cache_liveness_magnitude": "10", 2020-03-09T00:24:38.094 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_sessionmap_keys_per_op": "1024", 2020-03-09T00:24:38.095 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_shutdown_check": "0", 2020-03-09T00:24:38.095 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_skip_ino": "0", 2020-03-09T00:24:38.095 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_snap_max_uid": "4294967294", 2020-03-09T00:24:38.095 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_snap_min_uid": "0", 2020-03-09T00:24:38.095 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_snap_rstat": "false", 2020-03-09T00:24:38.095 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_task_status_update_interval": "2.000000", 2020-03-09T00:24:38.096 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_thrash_exports": "0", 2020-03-09T00:24:38.096 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_thrash_fragments": "0", 2020-03-09T00:24:38.096 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_tick_interval": "5.000000", 2020-03-09T00:24:38.096 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_verify_backtrace": "1", 2020-03-09T00:24:38.097 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_verify_scatter": "false", 2020-03-09T00:24:38.097 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_wipe_ino_prealloc": "false", 2020-03-09T00:24:38.097 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_wipe_sessions": "false", 2020-03-09T00:24:38.097 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool_debug": "false", 2020-03-09T00:24:38.097 INFO:teuthology.orchestra.run.smithi148.stdout: "memstore_debug_omit_block_device_write": "false", 2020-03-09T00:24:38.098 INFO:teuthology.orchestra.run.smithi148.stdout: "memstore_device_bytes": "1073741824", 2020-03-09T00:24:38.098 INFO:teuthology.orchestra.run.smithi148.stdout: "memstore_page_set": "false", 2020-03-09T00:24:38.098 INFO:teuthology.orchestra.run.smithi148.stdout: "memstore_page_size": "65536", 2020-03-09T00:24:38.098 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_client_bytes": "134217728", 2020-03-09T00:24:38.098 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_client_messages": "512", 2020-03-09T00:24:38.098 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2020-03-09T00:24:38.098 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_connect_retry_interval": "1.000000", 2020-03-09T00:24:38.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2020-03-09T00:24:38.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2020-03-09T00:24:38.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_initial_modules": "restful iostat", 2020-03-09T00:24:38.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_mds_bytes": "134217728", 2020-03-09T00:24:38.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_mds_messages": "128", 2020-03-09T00:24:38.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2020-03-09T00:24:38.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_mon_bytes": "134217728", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_mon_messages": "128", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_osd_bytes": "536870912", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_osd_messages": "8192", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_service_beacon_grace": "60.000000", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_stats_period": "5", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_stats_threshold": "5", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_tick_period": "2", 2020-03-09T00:24:38.100 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_accept_timeout_factor": "2.000000", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_allow_pool_delete": "true", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_bytes": "104857600", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_directed_command_retry": "5", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_hunt_interval": "3.000000", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2020-03-09T00:24:38.101 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2020-03-09T00:24:38.102 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2020-03-09T00:24:38.102 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_hunt_parallel": "3", 2020-03-09T00:24:38.102 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_max_log_entries_per_message": "1000", 2020-03-09T00:24:38.102 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_ping_interval": "10.000000", 2020-03-09T00:24:38.102 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_client_ping_timeout": "30.000000", 2020-03-09T00:24:38.102 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_clock_drift_allowed": "1.000000", 2020-03-09T00:24:38.102 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2020-03-09T00:24:38.103 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2020-03-09T00:24:38.103 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_file_level": "debug", 2020-03-09T00:24:38.103 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_file": "true", 2020-03-09T00:24:38.103 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_graylog": "false", 2020-03-09T00:24:38.103 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2020-03-09T00:24:38.103 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_graylog_port": "12201", 2020-03-09T00:24:38.103 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_stderr": "false", 2020-03-09T00:24:38.104 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_syslog": "default=false", 2020-03-09T00:24:38.104 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2020-03-09T00:24:38.104 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cluster_log_to_syslog_level": "info", 2020-03-09T00:24:38.104 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_compact_on_bootstrap": "false", 2020-03-09T00:24:38.104 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_compact_on_start": "false", 2020-03-09T00:24:38.104 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_compact_on_trim": "true", 2020-03-09T00:24:38.104 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_config_key_max_entry_size": "65536", 2020-03-09T00:24:38.105 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_cpu_threads": "4", 2020-03-09T00:24:38.105 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_crush_min_required_version": "hammer", 2020-03-09T00:24:38.105 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_daemon_bytes": "419430400", 2020-03-09T00:24:38.105 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2020-03-09T00:24:38.106 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_data_avail_crit": "5", 2020-03-09T00:24:38.106 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_data_avail_warn": "5", 2020-03-09T00:24:38.106 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_data_size_warn": "16106127360", 2020-03-09T00:24:38.106 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_block_osdmap_trim": "false", 2020-03-09T00:24:38.106 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_deprecated_as_obsolete": "false", 2020-03-09T00:24:38.106 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_dump_json": "false", 2020-03-09T00:24:38.107 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2020-03-09T00:24:38.107 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_dump_transactions": "false", 2020-03-09T00:24:38.107 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_extra_checks": "false", 2020-03-09T00:24:38.107 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_no_initial_persistent_features": "false", 2020-03-09T00:24:38.107 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2020-03-09T00:24:38.107 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_no_require_nautilus": "false", 2020-03-09T00:24:38.108 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_no_require_octopus": "false", 2020-03-09T00:24:38.108 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2020-03-09T00:24:38.108 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_delta_reset_interval": "10.000000", 2020-03-09T00:24:38.108 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_dns_srv_name": "ceph-mon", 2020-03-09T00:24:38.108 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_election_timeout": "5.000000", 2020-03-09T00:24:38.108 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_enable_op_tracker": "true", 2020-03-09T00:24:38.108 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_fake_pool_delete": "false", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_force_quorum_join": "false", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_globalid_prealloc": "10000", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_health_log_update_period": "5", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_health_max_detail": "50", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_health_to_clog": "true", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_health_to_clog_interval": "3600", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2020-03-09T00:24:38.109 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_host": "172.21.15.148,172.21.15.139,172.21.15.50", 2020-03-09T00:24:38.110 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_initial_members": "", 2020-03-09T00:24:38.110 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2020-03-09T00:24:38.110 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2020-03-09T00:24:38.110 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_inject_transaction_delay_max": "10.000000", 2020-03-09T00:24:38.110 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2020-03-09T00:24:38.110 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_keyvaluedb": "rocksdb", 2020-03-09T00:24:38.111 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_lease": "5.000000", 2020-03-09T00:24:38.111 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2020-03-09T00:24:38.111 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_lease_renew_interval_factor": "0.600000", 2020-03-09T00:24:38.111 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_log_max_summary": "50", 2020-03-09T00:24:38.111 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_log_entries_per_event": "4096", 2020-03-09T00:24:38.111 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_log_epochs": "500", 2020-03-09T00:24:38.112 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_mdsmap_epochs": "500", 2020-03-09T00:24:38.112 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_mgrmap_epochs": "500", 2020-03-09T00:24:38.112 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_osd": "10000", 2020-03-09T00:24:38.112 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_pg_per_osd": "10000", 2020-03-09T00:24:38.112 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_pool_pg_num": "65536", 2020-03-09T00:24:38.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_max_snap_prune_per_epoch": "100", 2020-03-09T00:24:38.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mds_blacklist_interval": "86400.000000", 2020-03-09T00:24:38.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mds_force_trim_to": "0", 2020-03-09T00:24:38.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mds_skip_sanity": "false", 2020-03-09T00:24:38.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_memory_autotune": "true", 2020-03-09T00:24:38.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_memory_target": "2147483648", 2020-03-09T00:24:38.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mgr_beacon_grace": "30", 2020-03-09T00:24:38.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mgr_blacklist_interval": "86400.000000", 2020-03-09T00:24:38.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mgr_digest_period": "5", 2020-03-09T00:24:38.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mgr_inactive_grace": "60", 2020-03-09T00:24:38.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mgr_mkfs_grace": "240", 2020-03-09T00:24:38.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2020-03-09T00:24:38.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_min_osdmap_epochs": "500", 2020-03-09T00:24:38.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_op_complaint_time": "30", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_op_history_duration": "600", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_op_history_size": "20", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_op_history_slow_op_size": "20", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_op_history_slow_op_threshold": "10", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_op_log_threshold": "5", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_adjust_down_out_interval": "true", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2020-03-09T00:24:38.115 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2020-03-09T00:24:38.116 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_auto_mark_in": "false", 2020-03-09T00:24:38.116 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_auto_mark_new_in": "true", 2020-03-09T00:24:38.116 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2020-03-09T00:24:38.116 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_blacklist_default_expire": "3600.000000", 2020-03-09T00:24:38.116 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_cache_size": "500", 2020-03-09T00:24:38.116 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_cache_size_min": "134217728", 2020-03-09T00:24:38.116 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_crush_smoke_test": "true", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_destroyed_out_interval": "600", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_down_out_interval": "600", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_down_out_subtree_limit": "rack", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_force_trim_to": "0", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_full_ratio": "0.950000", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_initial_require_min_compat_client": "jewel", 2020-03-09T00:24:38.117 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_laggy_halflife": "3600", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_laggy_max_interval": "300", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_laggy_weight": "0.300000", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_max_creating_pgs": "1024", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_max_initial_pgs": "1024", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_min_down_reporters": "2", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_min_in_ratio": "0.750000", 2020-03-09T00:24:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_min_up_ratio": "0.300000", 2020-03-09T00:24:38.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_nearfull_ratio": "0.850000", 2020-03-09T00:24:38.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_prime_pg_temp": "true", 2020-03-09T00:24:38.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2020-03-09T00:24:38.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2020-03-09T00:24:38.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_report_timeout": "900", 2020-03-09T00:24:38.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_reporter_subtree_level": "osd", 2020-03-09T00:24:38.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2020-03-09T00:24:38.120 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osd_warn_op_age": "32.000000", 2020-03-09T00:24:38.120 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osdmap_full_prune_enabled": "true", 2020-03-09T00:24:38.120 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osdmap_full_prune_interval": "10", 2020-03-09T00:24:38.120 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osdmap_full_prune_min": "10000", 2020-03-09T00:24:38.120 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_osdmap_full_prune_txsize": "100", 2020-03-09T00:24:38.120 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2020-03-09T00:24:38.120 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pg_stuck_threshold": "60", 2020-03-09T00:24:38.121 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2020-03-09T00:24:38.121 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pg_warn_min_objects": "10000", 2020-03-09T00:24:38.121 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pg_warn_min_per_osd": "0", 2020-03-09T00:24:38.121 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pg_warn_min_pool_objects": "1000", 2020-03-09T00:24:38.121 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pool_quota_crit_threshold": "0", 2020-03-09T00:24:38.121 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_pool_quota_warn_threshold": "0", 2020-03-09T00:24:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_probe_timeout": "2.000000", 2020-03-09T00:24:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_reweight_max_change": "0.050000", 2020-03-09T00:24:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_reweight_max_osds": "4", 2020-03-09T00:24:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_reweight_min_bytes_per_osd": "10", 2020-03-09T00:24:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_reweight_min_pgs_per_osd": "4", 2020-03-09T00:24:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2020-03-09T00:24:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2020-03-09T00:24:38.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2020-03-09T00:24:38.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_scrub_interval": "86400", 2020-03-09T00:24:38.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_scrub_max_keys": "100", 2020-03-09T00:24:38.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_scrub_timeout": "300", 2020-03-09T00:24:38.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_session_timeout": "300", 2020-03-09T00:24:38.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_smart_report_timeout": "5", 2020-03-09T00:24:38.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_stat_smooth_intervals": "6", 2020-03-09T00:24:38.124 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_subscribe_interval": "86400.000000", 2020-03-09T00:24:38.124 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_sync_debug": "false", 2020-03-09T00:24:38.124 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_sync_max_payload_keys": "2000", 2020-03-09T00:24:38.124 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_sync_max_payload_size": "1048576", 2020-03-09T00:24:38.124 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_sync_provider_kill_at": "0", 2020-03-09T00:24:38.124 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_sync_requester_kill_at": "0", 2020-03-09T00:24:38.124 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_sync_timeout": "60.000000", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_target_pg_per_osd": "100", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_tick_interval": "5", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_timecheck_interval": "300.000000", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_timecheck_skew_interval": "30.000000", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2020-03-09T00:24:38.125 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_misplaced": "false", 2020-03-09T00:24:38.126 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2020-03-09T00:24:38.126 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2020-03-09T00:24:38.126 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_pool_no_app": "true", 2020-03-09T00:24:38.126 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_pool_no_redundancy": "false", 2020-03-09T00:24:38.126 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2020-03-09T00:24:38.126 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2020-03-09T00:24:38.127 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2020-03-09T00:24:38.127 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_on_too_few_osds": "false", 2020-03-09T00:24:38.127 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2020-03-09T00:24:38.127 INFO:teuthology.orchestra.run.smithi148.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2020-03-09T00:24:38.127 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": "", 2020-03-09T00:24:38.127 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_max_op_threads": "5", 2020-03-09T00:24:38.128 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_op_threads": "3", 2020-03-09T00:24:38.128 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_buffer_size": "131072", 2020-03-09T00:24:38.128 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_cm": "false", 2020-03-09T00:24:38.128 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_device_name": "", 2020-03-09T00:24:38.128 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_dscp": "96", 2020-03-09T00:24:38.128 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_enable_hugepage": "false", 2020-03-09T00:24:38.128 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_gid_idx": "0", 2020-03-09T00:24:38.129 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_local_gid": "", 2020-03-09T00:24:38.129 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_polling_us": "1000", 2020-03-09T00:24:38.129 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_port_num": "1", 2020-03-09T00:24:38.129 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_receive_buffers": "32768", 2020-03-09T00:24:38.129 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_receive_queue_len": "4096", 2020-03-09T00:24:38.129 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_roce_ver": "1", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_send_buffers": "1024", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_sl": "3", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_support_srq": "true", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_async_rdma_type": "ib", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_before_connect": "false", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_ipv4": "true", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_ipv6": "false", 2020-03-09T00:24:38.130 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_msgr1": "true", 2020-03-09T00:24:38.131 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_msgr2": "true", 2020-03-09T00:24:38.131 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_port_max": "7300", 2020-03-09T00:24:38.131 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_port_min": "6800", 2020-03-09T00:24:38.131 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_prefer_ipv4": "false", 2020-03-09T00:24:38.131 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_retry_count": "3", 2020-03-09T00:24:38.131 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_bind_retry_delay": "5", 2020-03-09T00:24:38.132 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_blackhole_client": "false", 2020-03-09T00:24:38.132 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_blackhole_mds": "false", 2020-03-09T00:24:38.132 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_blackhole_mgr": "false", 2020-03-09T00:24:38.132 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_blackhole_mon": "false", 2020-03-09T00:24:38.132 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_blackhole_osd": "false", 2020-03-09T00:24:38.132 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_client_mode": "crc secure", 2020-03-09T00:24:38.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_cluster_mode": "crc secure", 2020-03-09T00:24:38.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_cluster_type": "", 2020-03-09T00:24:38.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_connection_idle_timeout": "900", 2020-03-09T00:24:38.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_connection_ready_timeout": "10", 2020-03-09T00:24:38.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_crc_data": "true", 2020-03-09T00:24:38.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_crc_header": "true", 2020-03-09T00:24:38.133 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_die_on_bad_msg": "false", 2020-03-09T00:24:38.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_die_on_bug": "true", 2020-03-09T00:24:38.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_die_on_old_message": "true", 2020-03-09T00:24:38.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_die_on_skipped_message": "false", 2020-03-09T00:24:38.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_die_on_unhandled_msg": "false", 2020-03-09T00:24:38.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dispatch_throttle_bytes": "104857600", 2020-03-09T00:24:38.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_coremask": "0xF", 2020-03-09T00:24:38.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_debug_allow_loopback": "false", 2020-03-09T00:24:38.135 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2020-03-09T00:24:38.135 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_host_ipv4_addr": "", 2020-03-09T00:24:38.135 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_hugepages": "", 2020-03-09T00:24:38.135 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_hw_flow_control": "true", 2020-03-09T00:24:38.135 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2020-03-09T00:24:38.135 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_lro": "true", 2020-03-09T00:24:38.135 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_memory_channel": "4", 2020-03-09T00:24:38.136 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2020-03-09T00:24:38.136 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_pmd": "", 2020-03-09T00:24:38.136 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_port_id": "0", 2020-03-09T00:24:38.136 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2020-03-09T00:24:38.136 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dump_corrupt_message_level": "1", 2020-03-09T00:24:38.137 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_dump_on_send": "false", 2020-03-09T00:24:38.137 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_initial_backoff": "0.200000", 2020-03-09T00:24:38.137 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_inject_delay_max": "1.000000", 2020-03-09T00:24:38.137 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_inject_delay_msg_type": "", 2020-03-09T00:24:38.137 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_inject_delay_probability": "0.000000", 2020-03-09T00:24:38.137 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_inject_delay_type": "", 2020-03-09T00:24:38.137 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_inject_internal_delays": "0.000000", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_inject_socket_failures": "2500", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_learn_addr_from_peer": "true", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_max_accept_failures": "4", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_max_backoff": "15.000000", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_mon_client_mode": "secure", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_mon_cluster_mode": "secure crc", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_mon_service_mode": "secure crc", 2020-03-09T00:24:38.138 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2020-03-09T00:24:38.139 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_pq_min_cost": "65536", 2020-03-09T00:24:38.139 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_public_type": "", 2020-03-09T00:24:38.139 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_service_mode": "crc secure", 2020-03-09T00:24:38.139 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_tcp_listen_backlog": "512", 2020-03-09T00:24:38.139 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_tcp_nodelay": "true", 2020-03-09T00:24:38.139 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_tcp_prefetch_max_size": "4096", 2020-03-09T00:24:38.139 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_tcp_rcvbuf": "0", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "ms_type": "async+posix", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "no_config_file": "false", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_completion_locks_per_session": "32", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_debug_inject_relock_delay": "false", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_inflight_op_bytes": "104857600", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_inflight_ops": "1024", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_inject_no_watch_ping": "false", 2020-03-09T00:24:38.140 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_retry_writes_after_first_reply": "false", 2020-03-09T00:24:38.141 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_tick_interval": "5.000000", 2020-03-09T00:24:38.141 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter_timeout": "10.000000", 2020-03-09T00:24:38.141 INFO:teuthology.orchestra.run.smithi148.stdout: "objectstore_blackhole": "false", 2020-03-09T00:24:38.141 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_agent_delay_time": "5.000000", 2020-03-09T00:24:38.141 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_agent_hist_halflife": "1000", 2020-03-09T00:24:38.141 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_agent_max_low_ops": "2", 2020-03-09T00:24:38.141 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_agent_max_ops": "4", 2020-03-09T00:24:38.142 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_agent_min_evict_effort": "0.100000", 2020-03-09T00:24:38.142 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_agent_quantize_effort": "0.100000", 2020-03-09T00:24:38.142 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_agent_slop": "0.020000", 2020-03-09T00:24:38.142 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_allow_recovery_below_min_size": "true", 2020-03-09T00:24:38.142 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_async_recovery_min_cost": "1", 2020-03-09T00:24:38.142 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_auto_mark_unfound_lost": "false", 2020-03-09T00:24:38.142 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_backfill_retry_interval": "30.000000", 2020-03-09T00:24:38.143 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_backfill_scan_max": "512", 2020-03-09T00:24:38.143 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_backfill_scan_min": "64", 2020-03-09T00:24:38.143 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_backoff_on_degraded": "false", 2020-03-09T00:24:38.143 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_backoff_on_peering": "false", 2020-03-09T00:24:38.143 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_backoff_on_unfound": "true", 2020-03-09T00:24:38.143 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_beacon_report_interval": "300", 2020-03-09T00:24:38.144 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bench_duration": "30", 2020-03-09T00:24:38.144 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bench_large_size_max_throughput": "104857600", 2020-03-09T00:24:38.144 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bench_max_block_size": "67108864", 2020-03-09T00:24:38.144 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bench_small_size_max_iops": "100", 2020-03-09T00:24:38.144 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_blkin_trace_all": "false", 2020-03-09T00:24:38.144 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2020-03-09T00:24:38.145 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2020-03-09T00:24:38.145 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_check_for_log_corruption": "false", 2020-03-09T00:24:38.145 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_check_max_object_name_len_on_startup": "true", 2020-03-09T00:24:38.145 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas", 2020-03-09T00:24:38.145 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2020-03-09T00:24:38.145 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas", 2020-03-09T00:24:38.145 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_class_update_on_start": "true", 2020-03-09T00:24:38.146 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_client_message_cap": "100", 2020-03-09T00:24:38.146 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_client_message_size_cap": "524288000", 2020-03-09T00:24:38.146 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_client_op_priority": "63", 2020-03-09T00:24:38.146 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_client_watch_timeout": "30", 2020-03-09T00:24:38.146 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_command_max_records": "256", 2020-03-09T00:24:38.146 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_command_thread_suicide_timeout": "900", 2020-03-09T00:24:38.146 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_command_thread_timeout": "600", 2020-03-09T00:24:38.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_copyfrom_max_chunk": "8388608", 2020-03-09T00:24:38.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_crush_chooseleaf_type": "0", 2020-03-09T00:24:38.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_crush_initial_weight": "-1.000000", 2020-03-09T00:24:38.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_crush_update_on_start": "true", 2020-03-09T00:24:38.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_crush_update_weight_set": "true", 2020-03-09T00:24:38.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2020-03-09T00:24:38.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2020-03-09T00:24:38.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2020-03-09T00:24:38.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_drop_ping_duration": "0", 2020-03-09T00:24:38.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_drop_ping_probability": "0.000000", 2020-03-09T00:24:38.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_feed_pullee": "-1", 2020-03-09T00:24:38.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_inject_copyfrom_error": "false", 2020-03-09T00:24:38.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2020-03-09T00:24:38.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2020-03-09T00:24:38.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_misdirected_ops": "false", 2020-03-09T00:24:38.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_no_acting_change": "false", 2020-03-09T00:24:38.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_no_purge_strays": "false", 2020-03-09T00:24:38.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_op_order": "false", 2020-03-09T00:24:38.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_pg_log_writeout": "false", 2020-03-09T00:24:38.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_pretend_recovery_active": "false", 2020-03-09T00:24:38.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_random_push_read_error": "0.000000", 2020-03-09T00:24:38.150 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2020-03-09T00:24:38.150 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_shutdown": "false", 2020-03-09T00:24:38.150 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2020-03-09T00:24:38.150 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2020-03-09T00:24:38.150 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_verify_cached_snaps": "false", 2020-03-09T00:24:38.150 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_verify_missing_on_start": "false", 2020-03-09T00:24:38.150 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_verify_snaps": "false", 2020-03-09T00:24:38.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_debug_verify_stray_on_activate": "false", 2020-03-09T00:24:38.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_deep_scrub_interval": "604800.000000", 2020-03-09T00:24:38.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_deep_scrub_keys": "1024", 2020-03-09T00:24:38.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2020-03-09T00:24:38.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2020-03-09T00:24:38.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2020-03-09T00:24:38.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_deep_scrub_stride": "524288", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_default_data_pool_replay_window": "5", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_default_notify_timeout": "30", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_delete_sleep": "0.000000", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_delete_sleep_hdd": "5.000000", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_delete_sleep_hybrid": "2.000000", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_delete_sleep_ssd": "0.000000", 2020-03-09T00:24:38.152 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_discard_disconnected_ops": "true", 2020-03-09T00:24:38.153 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_enable_op_tracker": "true", 2020-03-09T00:24:38.153 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2020-03-09T00:24:38.153 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_failsafe_full_ratio": "0.970000", 2020-03-09T00:24:38.153 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_fast_fail_on_connection_refused": "true", 2020-03-09T00:24:38.153 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_fast_info": "true", 2020-03-09T00:24:38.153 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_fast_shutdown": "true", 2020-03-09T00:24:38.153 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_find_best_info_ignore_history_les": "false", 2020-03-09T00:24:38.154 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_force_auth_primary_missing_objects": "100", 2020-03-09T00:24:38.154 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2020-03-09T00:24:38.154 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_function_tracing": "false", 2020-03-09T00:24:38.154 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_heartbeat_grace": "20", 2020-03-09T00:24:38.154 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_heartbeat_interval": "6", 2020-03-09T00:24:38.154 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2020-03-09T00:24:38.154 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_heartbeat_min_peers": "10", 2020-03-09T00:24:38.155 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_heartbeat_min_size": "2000", 2020-03-09T00:24:38.155 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_heartbeat_stale": "600", 2020-03-09T00:24:38.155 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2020-03-09T00:24:38.155 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_hit_set_max_size": "100000", 2020-03-09T00:24:38.156 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_hit_set_min_size": "1000", 2020-03-09T00:24:38.156 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_hit_set_namespace": ".ceph-internal", 2020-03-09T00:24:38.156 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_ignore_stale_divergent_priors": "false", 2020-03-09T00:24:38.156 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2020-03-09T00:24:38.156 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_inject_failure_on_pg_removal": "false", 2020-03-09T00:24:38.157 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2020-03-09T00:24:38.157 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_journal_flush_on_shutdown": "true", 2020-03-09T00:24:38.157 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_journal_size": "5120", 2020-03-09T00:24:38.157 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_kill_backfill_at": "0", 2020-03-09T00:24:38.157 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_loop_before_reset_tphandle": "64", 2020-03-09T00:24:38.157 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_map_cache_size": "50", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_map_dedup": "true", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_map_message_max": "40", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_map_message_max_bytes": "10485760", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_map_share_max_epochs": "40", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_attr_name_len": "100", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_attr_size": "0", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_backfills": "1", 2020-03-09T00:24:38.158 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_markdown_count": "5", 2020-03-09T00:24:38.159 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_markdown_period": "600", 2020-03-09T00:24:38.159 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_object_name_len": "2048", 2020-03-09T00:24:38.159 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_object_namespace_len": "256", 2020-03-09T00:24:38.159 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_object_size": "134217728", 2020-03-09T00:24:38.159 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2020-03-09T00:24:38.159 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_omap_entries_per_request": "1024", 2020-03-09T00:24:38.159 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_pg_blocked_by": "16", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_pg_log_entries": "10000", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_pgls": "1024", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_push_cost": "8388608", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_push_objects": "10", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_scrubs": "1", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2020-03-09T00:24:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_trimming_pgs": "2", 2020-03-09T00:24:38.161 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_write_op_reply_len": "32", 2020-03-09T00:24:38.161 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_max_write_size": "90", 2020-03-09T00:24:38.161 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2020-03-09T00:24:38.161 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2020-03-09T00:24:38.161 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2020-03-09T00:24:38.161 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2020-03-09T00:24:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2020-03-09T00:24:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2020-03-09T00:24:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2020-03-09T00:24:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_client_lim": "999999", 2020-03-09T00:24:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_client_res": "1", 2020-03-09T00:24:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mclock_scheduler_client_wgt": "1", 2020-03-09T00:24:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_memory_base": "805306368", 2020-03-09T00:24:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_memory_cache_min": "134217728", 2020-03-09T00:24:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_memory_cache_resize_interval": "1.000000", 2020-03-09T00:24:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_memory_expected_fragmentation": "0.150000", 2020-03-09T00:24:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_memory_target": "4294967296", 2020-03-09T00:24:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2020-03-09T00:24:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_min_pg_log_entries": "250", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_min_recovery_priority": "0", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mon_ack_timeout": "30.000000", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mon_heartbeat_interval": "30", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mon_report_interval": "5", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mon_report_max_in_flight": "2", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mon_shutdown_timeout": "5.000000", 2020-03-09T00:24:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_num_cache_shards": "32", 2020-03-09T00:24:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_num_op_tracker_shard": "32", 2020-03-09T00:24:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_numa_auto_affinity": "true", 2020-03-09T00:24:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_numa_node": "-1", 2020-03-09T00:24:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_numa_prefer_iface": "true", 2020-03-09T00:24:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_object_clean_region_max_num_intervals": "10", 2020-03-09T00:24:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_objecter_finishers": "1", 2020-03-09T00:24:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_objectstore": "bluestore", 2020-03-09T00:24:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_objectstore_fuse": "false", 2020-03-09T00:24:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_objectstore_tracing": "false", 2020-03-09T00:24:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_complaint_time": "30.000000", 2020-03-09T00:24:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_history_duration": "600", 2020-03-09T00:24:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_history_size": "20", 2020-03-09T00:24:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_history_slow_op_size": "20", 2020-03-09T00:24:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_log_threshold": "5", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_num_shards": "0", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_num_shards_hdd": "5", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_num_shards_ssd": "8", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_num_threads_per_shard": "0", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2020-03-09T00:24:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2020-03-09T00:24:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_pq_min_cost": "65536", 2020-03-09T00:24:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_queue": "wpq", 2020-03-09T00:24:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_queue_cut_off": "high", 2020-03-09T00:24:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_thread_suicide_timeout": "150", 2020-03-09T00:24:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_op_thread_timeout": "15", 2020-03-09T00:24:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_open_classes_on_start": "true", 2020-03-09T00:24:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_os_flags": "0", 2020-03-09T00:24:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_peering_op_priority": "255", 2020-03-09T00:24:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_delete_cost": "1048576", 2020-03-09T00:24:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_delete_priority": "5", 2020-03-09T00:24:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2020-03-09T00:24:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2020-03-09T00:24:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_log_dups_tracked": "3000", 2020-03-09T00:24:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_log_trim_max": "10000", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_log_trim_min": "100", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_object_context_cache_count": "64", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pg_stat_report_interval_max": "500", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_cache_min_evict_age": "0", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_cache_min_flush_age": "0", 2020-03-09T00:24:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2020-03-09T00:24:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2020-03-09T00:24:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2020-03-09T00:24:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_crush_rule": "-1", 2020-03-09T00:24:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_ec_fast_read": "false", 2020-03-09T00:24:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2020-03-09T00:24:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_flag_hashpspool": "true", 2020-03-09T00:24:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_flag_nodelete": "false", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_flag_nopgchange": "false", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_flag_nosizechange": "false", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_flags": "0", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_min_size": "0", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_pg_num": "32", 2020-03-09T00:24:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_pgp_num": "0", 2020-03-09T00:24:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2020-03-09T00:24:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_size": "2", 2020-03-09T00:24:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_default_type": "replicated", 2020-03-09T00:24:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2020-03-09T00:24:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pool_use_gmt_hitset": "true", 2020-03-09T00:24:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_push_per_object_cost": "1000", 2020-03-09T00:24:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_read_ec_check_for_errors": "false", 2020-03-09T00:24:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recover_clone_overlap": "true", 2020-03-09T00:24:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recover_clone_overlap_limit": "10", 2020-03-09T00:24:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_cost": "20971520", 2020-03-09T00:24:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_delay_start": "0.000000", 2020-03-09T00:24:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_max_active": "0", 2020-03-09T00:24:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_max_active_hdd": "3", 2020-03-09T00:24:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_max_active_ssd": "10", 2020-03-09T00:24:38.176 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_max_chunk": "8388608", 2020-03-09T00:24:38.176 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2020-03-09T00:24:38.176 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_max_single_start": "1", 2020-03-09T00:24:38.177 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_op_priority": "3", 2020-03-09T00:24:38.177 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_op_warn_multiple": "16", 2020-03-09T00:24:38.177 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_priority": "5", 2020-03-09T00:24:38.177 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_retry_interval": "30.000000", 2020-03-09T00:24:38.178 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_sleep": "0.000000", 2020-03-09T00:24:38.178 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_sleep_hdd": "0.100000", 2020-03-09T00:24:38.178 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2020-03-09T00:24:38.178 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_recovery_sleep_ssd": "0.000000", 2020-03-09T00:24:38.179 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_repair_during_recovery": "false", 2020-03-09T00:24:38.179 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_requested_scrub_priority": "120", 2020-03-09T00:24:38.179 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_rollback_to_cluster_snap": "", 2020-03-09T00:24:38.179 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_auto_repair": "false", 2020-03-09T00:24:38.179 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_auto_repair_num_errors": "5", 2020-03-09T00:24:38.180 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_backoff_ratio": "0.660000", 2020-03-09T00:24:38.180 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_begin_hour": "0", 2020-03-09T00:24:38.180 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_begin_week_day": "0", 2020-03-09T00:24:38.180 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_chunk_max": "25", 2020-03-09T00:24:38.180 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_chunk_min": "5", 2020-03-09T00:24:38.180 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_cost": "52428800", 2020-03-09T00:24:38.180 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_during_recovery": "false", 2020-03-09T00:24:38.181 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_end_hour": "24", 2020-03-09T00:24:38.181 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_end_week_day": "7", 2020-03-09T00:24:38.181 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_extended_sleep": "0.000000", 2020-03-09T00:24:38.181 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2020-03-09T00:24:38.181 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_invalid_stats": "true", 2020-03-09T00:24:38.182 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_load_threshold": "0.500000", 2020-03-09T00:24:38.182 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_max_interval": "604800.000000", 2020-03-09T00:24:38.182 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_max_preemptions": "5", 2020-03-09T00:24:38.182 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_min_interval": "86400.000000", 2020-03-09T00:24:38.183 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_priority": "5", 2020-03-09T00:24:38.183 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_scrub_sleep": "0.000000", 2020-03-09T00:24:38.183 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_shutdown_pgref_assert": "false", 2020-03-09T00:24:38.183 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_skip_data_digest": "false", 2020-03-09T00:24:38.183 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_smart_report_timeout": "5", 2020-03-09T00:24:38.184 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_snap_trim_cost": "1048576", 2020-03-09T00:24:38.184 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_snap_trim_priority": "5", 2020-03-09T00:24:38.184 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_snap_trim_sleep": "0.000000", 2020-03-09T00:24:38.184 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2020-03-09T00:24:38.185 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2020-03-09T00:24:38.185 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2020-03-09T00:24:38.185 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_stats_ack_timeout_decay": "0.900000", 2020-03-09T00:24:38.185 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_stats_ack_timeout_factor": "2.000000", 2020-03-09T00:24:38.186 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2020-03-09T00:24:38.186 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_target_transaction_size": "30", 2020-03-09T00:24:38.186 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_hit_set_count": "4", 2020-03-09T00:24:38.186 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2020-03-09T00:24:38.187 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2020-03-09T00:24:38.187 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2020-03-09T00:24:38.187 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2020-03-09T00:24:38.187 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2020-03-09T00:24:38.187 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2020-03-09T00:24:38.188 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_default_cache_mode": "writeback", 2020-03-09T00:24:38.188 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2020-03-09T00:24:38.188 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tier_promote_max_objects_sec": "25", 2020-03-09T00:24:38.188 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_tracing": "false", 2020-03-09T00:24:38.189 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_use_stale_snap": "false", 2020-03-09T00:24:38.189 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2020-03-09T00:24:38.189 INFO:teuthology.orchestra.run.smithi148.stdout: "osdc_blkin_trace_all": "false", 2020-03-09T00:24:38.189 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_kill_at": "0", 2020-03-09T00:24:38.190 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_max_join_drift": "10", 2020-03-09T00:24:38.190 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_min": "500", 2020-03-09T00:24:38.190 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_min_wait": "0.050000", 2020-03-09T00:24:38.190 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_propose_interval": "1.000000", 2020-03-09T00:24:38.190 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_service_trim_max": "500", 2020-03-09T00:24:38.191 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_service_trim_min": "250", 2020-03-09T00:24:38.191 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_stash_full_interval": "25", 2020-03-09T00:24:38.191 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_trim_max": "500", 2020-03-09T00:24:38.191 INFO:teuthology.orchestra.run.smithi148.stdout: "paxos_trim_min": "250", 2020-03-09T00:24:38.192 INFO:teuthology.orchestra.run.smithi148.stdout: "perf": "true", 2020-03-09T00:24:38.192 INFO:teuthology.orchestra.run.smithi148.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2020-03-09T00:24:38.192 INFO:teuthology.orchestra.run.smithi148.stdout: "plugin_crypto_accelerator": "crypto_isal", 2020-03-09T00:24:38.192 INFO:teuthology.orchestra.run.smithi148.stdout: "plugin_dir": "/usr/lib64/ceph", 2020-03-09T00:24:38.192 INFO:teuthology.orchestra.run.smithi148.stdout: "prompt": "\u001b[01;33mCephFS:~\u001b[96m/\u001b[0m\u001b[01;33m>>>\u001b[00m ", 2020-03-09T00:24:38.193 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "-", 2020-03-09T00:24:38.193 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrv": "", 2020-03-09T00:24:38.193 INFO:teuthology.orchestra.run.smithi148.stdout: "public_bind_addr": "-", 2020-03-09T00:24:38.193 INFO:teuthology.orchestra.run.smithi148.stdout: "public_network": "", 2020-03-09T00:24:38.194 INFO:teuthology.orchestra.run.smithi148.stdout: "public_network_interface": "", 2020-03-09T00:24:38.194 INFO:teuthology.orchestra.run.smithi148.stdout: "qat_compressor_enabled": "false", 2020-03-09T00:24:38.194 INFO:teuthology.orchestra.run.smithi148.stdout: "quiet": "false", 2020-03-09T00:24:38.195 INFO:teuthology.orchestra.run.smithi148.stdout: "rados_mon_op_timeout": "0.000000", 2020-03-09T00:24:38.195 INFO:teuthology.orchestra.run.smithi148.stdout: "rados_osd_op_timeout": "0.000000", 2020-03-09T00:24:38.195 INFO:teuthology.orchestra.run.smithi148.stdout: "rados_tracing": "false", 2020-03-09T00:24:38.196 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_atime_update_interval": "60", 2020-03-09T00:24:38.196 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2020-03-09T00:24:38.196 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_balance_parent_reads": "false", 2020-03-09T00:24:38.196 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_balance_snap_reads": "false", 2020-03-09T00:24:38.196 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_blacklist_expire_seconds": "0", 2020-03-09T00:24:38.197 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_blacklist_on_break_lock": "true", 2020-03-09T00:24:38.197 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_blkin_trace_all": "false", 2020-03-09T00:24:38.197 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache": "true", 2020-03-09T00:24:38.197 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_block_writes_upfront": "false", 2020-03-09T00:24:38.197 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_max_dirty": "25165824", 2020-03-09T00:24:38.198 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_max_dirty_age": "1.000000", 2020-03-09T00:24:38.198 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_max_dirty_object": "0", 2020-03-09T00:24:38.198 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_policy": "writearound", 2020-03-09T00:24:38.198 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_size": "33554432", 2020-03-09T00:24:38.198 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_target_dirty": "16777216", 2020-03-09T00:24:38.198 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_cache_writethrough_until_flush": "true", 2020-03-09T00:24:38.199 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_clone_copy_on_read": "false", 2020-03-09T00:24:38.199 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_compression_hint": "none", 2020-03-09T00:24:38.199 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_concurrent_management_ops": "10", 2020-03-09T00:24:38.199 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_clone_format": "auto", 2020-03-09T00:24:38.199 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_data_pool": "", 2020-03-09T00:24:38.199 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_features": "61", 2020-03-09T00:24:38.199 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_format": "2", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_map_options": "", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_order": "22", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_pool": "rbd", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_stripe_count": "0", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_default_stripe_unit": "0", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_disable_zero_copy_writes": "true", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_discard_granularity_bytes": "65536", 2020-03-09T00:24:38.200 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_discard_on_zeroed_write_same": "true", 2020-03-09T00:24:38.201 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_enable_alloc_hint": "true", 2020-03-09T00:24:38.201 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_io_scheduler": "simple", 2020-03-09T00:24:38.201 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2020-03-09T00:24:38.201 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_commit_age": "5.000000", 2020-03-09T00:24:38.201 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2020-03-09T00:24:38.201 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_max_payload_bytes": "16384", 2020-03-09T00:24:38.201 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_object_flush_age": "0.000000", 2020-03-09T00:24:38.202 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_object_flush_bytes": "1048576", 2020-03-09T00:24:38.202 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_object_flush_interval": "0", 2020-03-09T00:24:38.202 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2020-03-09T00:24:38.202 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2020-03-09T00:24:38.202 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_order": "24", 2020-03-09T00:24:38.202 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_pool": "", 2020-03-09T00:24:38.202 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_journal_splay_width": "4", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_localize_parent_reads": "false", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_localize_snap_reads": "false", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2020-03-09T00:24:38.203 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2020-03-09T00:24:38.204 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_image_policy_type": "simple", 2020-03-09T00:24:38.204 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2020-03-09T00:24:38.204 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_image_state_check_interval": "30", 2020-03-09T00:24:38.204 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2020-03-09T00:24:38.204 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2020-03-09T00:24:38.204 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2020-03-09T00:24:38.204 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2020-03-09T00:24:38.205 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2020-03-09T00:24:38.205 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_memory_autotune": "true", 2020-03-09T00:24:38.205 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_memory_base": "805306368", 2020-03-09T00:24:38.205 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2020-03-09T00:24:38.205 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_memory_cache_min": "134217728", 2020-03-09T00:24:38.205 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2020-03-09T00:24:38.206 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2020-03-09T00:24:38.206 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_memory_target": "4294967296", 2020-03-09T00:24:38.206 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_perf_stats_prio": "5", 2020-03-09T00:24:38.206 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2020-03-09T00:24:38.206 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2020-03-09T00:24:38.206 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirroring_delete_delay": "0", 2020-03-09T00:24:38.206 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirroring_max_mirroring_snapshots": "3", 2020-03-09T00:24:38.207 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirroring_replay_delay": "0", 2020-03-09T00:24:38.207 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2020-03-09T00:24:38.207 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2020-03-09T00:24:38.207 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_move_to_trash_on_remove": "false", 2020-03-09T00:24:38.207 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2020-03-09T00:24:38.207 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_mtime_update_interval": "60", 2020-03-09T00:24:38.208 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_non_blocking_aio": "true", 2020-03-09T00:24:38.208 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_op_thread_timeout": "60", 2020-03-09T00:24:38.208 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_op_threads": "1", 2020-03-09T00:24:38.208 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_parent_cache_enabled": "false", 2020-03-09T00:24:38.208 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_bps_burst": "0", 2020-03-09T00:24:38.208 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_bps_limit": "0", 2020-03-09T00:24:38.208 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_iops_burst": "0", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_iops_limit": "0", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_read_bps_burst": "0", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_read_bps_limit": "0", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_read_iops_burst": "0", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_read_iops_limit": "0", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_schedule_tick_min": "50", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_write_bps_burst": "0", 2020-03-09T00:24:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_write_bps_limit": "0", 2020-03-09T00:24:38.210 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_write_iops_burst": "0", 2020-03-09T00:24:38.210 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_qos_write_iops_limit": "0", 2020-03-09T00:24:38.210 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_read_from_replica_policy": "default", 2020-03-09T00:24:38.210 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2020-03-09T00:24:38.210 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_readahead_max_bytes": "524288", 2020-03-09T00:24:38.210 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_readahead_trigger_requests": "10", 2020-03-09T00:24:38.210 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_request_timed_out_seconds": "30", 2020-03-09T00:24:38.211 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_rwl_enabled": "false", 2020-03-09T00:24:38.211 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_rwl_log_periodic_stats": "false", 2020-03-09T00:24:38.211 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_rwl_path": "/tmp", 2020-03-09T00:24:38.211 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_rwl_size": "1073741824", 2020-03-09T00:24:38.211 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_skip_partial_discard": "true", 2020-03-09T00:24:38.211 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2020-03-09T00:24:38.211 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_tracing": "false", 2020-03-09T00:24:38.212 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_validate_names": "true", 2020-03-09T00:24:38.212 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd_validate_pool": "true", 2020-03-09T00:24:38.212 INFO:teuthology.orchestra.run.smithi148.stdout: "restapi_base_url": "", 2020-03-09T00:24:38.212 INFO:teuthology.orchestra.run.smithi148.stdout: "restapi_log_level": "", 2020-03-09T00:24:38.212 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_acl_grants_max_num": "100", 2020-03-09T00:24:38.212 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_admin_entry": "admin", 2020-03-09T00:24:38.213 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_barbican_url": "", 2020-03-09T00:24:38.213 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2020-03-09T00:24:38.213 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_bucket_default_quota_max_size": "-1", 2020-03-09T00:24:38.213 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_bucket_index_max_aio": "128", 2020-03-09T00:24:38.213 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_bucket_quota_cache_size": "10000", 2020-03-09T00:24:38.214 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_bucket_quota_soft_threshold": "0.950000", 2020-03-09T00:24:38.214 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_bucket_quota_ttl": "600", 2020-03-09T00:24:38.214 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_cache_enabled": "true", 2020-03-09T00:24:38.214 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_cache_expiry_interval": "900", 2020-03-09T00:24:38.214 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_cache_lru_size": "10000", 2020-03-09T00:24:38.214 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_content_length_compat": "false", 2020-03-09T00:24:38.214 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_copy_obj_progress": "true", 2020-03-09T00:24:38.215 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2020-03-09T00:24:38.215 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_cors_rules_max_num": "100", 2020-03-09T00:24:38.215 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_cross_domain_policy": "", 2020-03-09T00:24:38.215 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_default_encryption_key": "", 2020-03-09T00:24:38.215 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_require_ssl": "true", 2020-03-09T00:24:38.215 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2020-03-09T00:24:38.215 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_suppress_logs": "true", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_vault_addr": "", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_vault_auth": "token", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_vault_namespace": "", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_vault_prefix": "", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_vault_secret_engine": "transit", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_crypt_vault_token_file": "", 2020-03-09T00:24:38.216 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_curl_low_speed_limit": "1024", 2020-03-09T00:24:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_curl_low_speed_time": "300", 2020-03-09T00:24:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_curl_wait_timeout_ms": "1000", 2020-03-09T00:24:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2020-03-09T00:24:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_data_log_changes_size": "1000", 2020-03-09T00:24:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_data_log_num_shards": "128", 2020-03-09T00:24:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_data_log_obj_prefix": "data_log", 2020-03-09T00:24:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_data_log_window": "30", 2020-03-09T00:24:38.218 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_data_notify_interval_msec": "200", 2020-03-09T00:24:38.218 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_default_realm_info_oid": "default.realm", 2020-03-09T00:24:38.218 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_default_region_info_oid": "default.region", 2020-03-09T00:24:38.218 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_default_zone_info_oid": "default.zone", 2020-03-09T00:24:38.218 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2020-03-09T00:24:38.218 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_defer_to_bucket_acls": "", 2020-03-09T00:24:38.218 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_delete_multi_obj_max_num": "1000", 2020-03-09T00:24:38.219 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_admin_lim": "0.000000", 2020-03-09T00:24:38.219 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_admin_res": "100.000000", 2020-03-09T00:24:38.219 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2020-03-09T00:24:38.219 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_auth_lim": "0.000000", 2020-03-09T00:24:38.219 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_auth_res": "200.000000", 2020-03-09T00:24:38.219 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2020-03-09T00:24:38.219 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_data_lim": "0.000000", 2020-03-09T00:24:38.220 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_data_res": "500.000000", 2020-03-09T00:24:38.220 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_data_wgt": "500.000000", 2020-03-09T00:24:38.220 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2020-03-09T00:24:38.220 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_metadata_res": "500.000000", 2020-03-09T00:24:38.220 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2020-03-09T00:24:38.220 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dns_name": "", 2020-03-09T00:24:38.220 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dns_s3website_name": "", 2020-03-09T00:24:38.221 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_dynamic_resharding": "true", 2020-03-09T00:24:38.221 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, pubsub", 2020-03-09T00:24:38.221 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enable_gc_threads": "true", 2020-03-09T00:24:38.221 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enable_lc_threads": "true", 2020-03-09T00:24:38.221 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enable_ops_log": "false", 2020-03-09T00:24:38.221 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enable_quota_threads": "true", 2020-03-09T00:24:38.222 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enable_static_website": "false", 2020-03-09T00:24:38.222 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enable_usage_log": "false", 2020-03-09T00:24:38.222 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_enforce_swift_acls": "true", 2020-03-09T00:24:38.222 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_exit_timeout_secs": "120", 2020-03-09T00:24:38.222 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_expose_bucket": "false", 2020-03-09T00:24:38.222 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_extended_http_attrs": "", 2020-03-09T00:24:38.222 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_fcgi_socket_backlog": "1024", 2020-03-09T00:24:38.223 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2020-03-09T00:24:38.223 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_frontends": "beast port=7480", 2020-03-09T00:24:38.223 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_max_concurrent_io": "10", 2020-03-09T00:24:38.223 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_max_deferred": "50", 2020-03-09T00:24:38.223 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2020-03-09T00:24:38.223 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_max_objs": "32", 2020-03-09T00:24:38.224 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_max_queue_size": "134213632", 2020-03-09T00:24:38.224 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_max_trim_chunk": "16", 2020-03-09T00:24:38.224 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_obj_min_wait": "7200", 2020-03-09T00:24:38.224 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_processor_max_time": "3600", 2020-03-09T00:24:38.224 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_gc_processor_period": "3600", 2020-03-09T00:24:38.224 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_get_obj_max_req_size": "4194304", 2020-03-09T00:24:38.224 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_get_obj_window_size": "16777216", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_healthcheck_disabling_path": "", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_host": "", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ignore_get_invalid_range": "false", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_init_timeout": "300", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_accepted_admin_roles": "", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2020-03-09T00:24:38.225 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_domain": "", 2020-03-09T00:24:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_password": "", 2020-03-09T00:24:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_password_path": "", 2020-03-09T00:24:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_project": "", 2020-03-09T00:24:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_tenant": "", 2020-03-09T00:24:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_token": "", 2020-03-09T00:24:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_token_path": "", 2020-03-09T00:24:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_admin_user": "", 2020-03-09T00:24:38.227 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_api_version": "2", 2020-03-09T00:24:38.227 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_barbican_domain": "", 2020-03-09T00:24:38.227 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_barbican_password": "", 2020-03-09T00:24:38.227 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_barbican_project": "", 2020-03-09T00:24:38.227 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_barbican_tenant": "", 2020-03-09T00:24:38.227 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_barbican_user": "", 2020-03-09T00:24:38.227 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_implicit_tenants": "false", 2020-03-09T00:24:38.228 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_token_cache_size": "10000", 2020-03-09T00:24:38.228 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_url": "", 2020-03-09T00:24:38.228 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_keystone_verify_ssl": "true", 2020-03-09T00:24:38.228 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_lc_debug_interval": "-1", 2020-03-09T00:24:38.228 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_lc_lock_max_time": "60", 2020-03-09T00:24:38.229 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_lc_max_objs": "32", 2020-03-09T00:24:38.229 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_lc_max_rules": "1000", 2020-03-09T00:24:38.229 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_lc_thread_delay": "0", 2020-03-09T00:24:38.229 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2020-03-09T00:24:38.230 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ldap_dnattr": "uid", 2020-03-09T00:24:38.230 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2020-03-09T00:24:38.230 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ldap_searchfilter": "", 2020-03-09T00:24:38.230 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2020-03-09T00:24:38.230 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ldap_uri": "ldaps://", 2020-03-09T00:24:38.231 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2020-03-09T00:24:38.231 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_list_bucket_min_readahead": "1000", 2020-03-09T00:24:38.231 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_list_buckets_max_chunk": "1000", 2020-03-09T00:24:38.231 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_log_http_headers": "", 2020-03-09T00:24:38.232 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_log_nonexistent_bucket": "false", 2020-03-09T00:24:38.232 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2020-03-09T00:24:38.232 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_log_object_name_utc": "false", 2020-03-09T00:24:38.232 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_attr_name_len": "0", 2020-03-09T00:24:38.232 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_attr_size": "0", 2020-03-09T00:24:38.232 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_attrs_num_in_req": "0", 2020-03-09T00:24:38.233 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_chunk_size": "4194304", 2020-03-09T00:24:38.233 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_concurrent_requests": "1024", 2020-03-09T00:24:38.233 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_dynamic_shards": "1999", 2020-03-09T00:24:38.233 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_listing_results": "1000", 2020-03-09T00:24:38.233 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_notify_retries": "3", 2020-03-09T00:24:38.233 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_objs_per_shard": "100000", 2020-03-09T00:24:38.234 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_put_param_size": "1048576", 2020-03-09T00:24:38.234 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_put_size": "5368709120", 2020-03-09T00:24:38.234 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_max_slo_entries": "1000", 2020-03-09T00:24:38.234 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_md_log_max_shards": "64", 2020-03-09T00:24:38.234 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_md_notify_interval_msec": "200", 2020-03-09T00:24:38.234 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_mime_types_file": "/etc/mime.types", 2020-03-09T00:24:38.234 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_mp_lock_max_time": "600", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_multipart_min_part_size": "5242880", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_multipart_part_upload_limit": "10000", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_fhcache_partitions": "3", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_fhcache_size": "2017", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_lru_lanes": "5", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_max_gc": "300", 2020-03-09T00:24:38.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_namespace_expire_secs": "300", 2020-03-09T00:24:38.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_s3_fast_attrs": "false", 2020-03-09T00:24:38.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_nfs_write_completion_interval_s": "10", 2020-03-09T00:24:38.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_num_async_rados_threads": "32", 2020-03-09T00:24:38.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_num_control_oids": "8", 2020-03-09T00:24:38.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_num_rados_handles": "1", 2020-03-09T00:24:38.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_numa_node": "-1", 2020-03-09T00:24:38.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_obj_stripe_size": "4194304", 2020-03-09T00:24:38.237 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_obj_tombstone_cache_size": "1000", 2020-03-09T00:24:38.237 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_objexp_chunk_size": "100", 2020-03-09T00:24:38.237 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_objexp_gc_interval": "600", 2020-03-09T00:24:38.237 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_objexp_hints_num_shards": "127", 2020-03-09T00:24:38.237 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_olh_pending_timeout_sec": "3600", 2020-03-09T00:24:38.237 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_op_thread_suicide_timeout": "0", 2020-03-09T00:24:38.237 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_op_thread_timeout": "600", 2020-03-09T00:24:38.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_op_tracing": "false", 2020-03-09T00:24:38.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_opa_token": "", 2020-03-09T00:24:38.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_opa_url": "", 2020-03-09T00:24:38.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_opa_verify_ssl": "true", 2020-03-09T00:24:38.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ops_log_data_backlog": "5242880", 2020-03-09T00:24:38.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ops_log_rados": "true", 2020-03-09T00:24:38.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_ops_log_socket_path": "", 2020-03-09T00:24:38.239 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_override_bucket_index_max_shards": "0", 2020-03-09T00:24:38.239 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2020-03-09T00:24:38.239 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_period_push_interval": "2.000000", 2020-03-09T00:24:38.239 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_period_push_interval_max": "30.000000", 2020-03-09T00:24:38.240 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_period_root_pool": ".rgw.root", 2020-03-09T00:24:38.240 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_port": "", 2020-03-09T00:24:38.240 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_print_continue": "true", 2020-03-09T00:24:38.240 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_print_prohibited_content_length": "false", 2020-03-09T00:24:38.240 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_put_obj_max_window_size": "67108864", 2020-03-09T00:24:38.241 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_put_obj_min_window_size": "16777216", 2020-03-09T00:24:38.241 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2020-03-09T00:24:38.241 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_rados_pool_pg_num_min": "8", 2020-03-09T00:24:38.241 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_rados_pool_recovery_priority": "5", 2020-03-09T00:24:38.242 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_rados_tracing": "false", 2020-03-09T00:24:38.242 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_realm": "", 2020-03-09T00:24:38.242 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_realm_root_pool": ".rgw.root", 2020-03-09T00:24:38.242 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_region": "", 2020-03-09T00:24:38.242 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_region_root_pool": ".rgw.root", 2020-03-09T00:24:38.242 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_relaxed_region_enforcement": "false", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_relaxed_s3_bucket_names": "false", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_request_uri": "", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_reshard_batch_size": "64", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_reshard_bucket_lock_duration": "360", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_reshard_max_aio": "128", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_reshard_num_logs": "16", 2020-03-09T00:24:38.243 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_reshard_thread_interval": "600", 2020-03-09T00:24:38.244 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_resolve_cname": "false", 2020-03-09T00:24:38.244 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_rest_getusage_op_compat": "false", 2020-03-09T00:24:38.244 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_run_sync_thread": "true", 2020-03-09T00:24:38.244 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_s3_auth_order": "sts, external, local", 2020-03-09T00:24:38.244 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_s3_auth_use_keystone": "false", 2020-03-09T00:24:38.244 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_s3_auth_use_ldap": "false", 2020-03-09T00:24:38.244 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_s3_auth_use_rados": "true", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_s3_auth_use_sts": "false", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_s3_success_create_obj_status": "0", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_safe_max_objects_per_shard": "102400", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_scheduler_type": "throttler", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_script_uri": "", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_service_provider_name": "", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_shard_warning_threshold": "90.000000", 2020-03-09T00:24:38.245 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_socket_path": "", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sts_client_id": "", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sts_client_secret": "", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sts_entry": "sts", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sts_key": "sts", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sts_max_session_duration": "43200", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sts_token_introspection_url": "", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_account_in_url": "false", 2020-03-09T00:24:38.246 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_auth_entry": "auth", 2020-03-09T00:24:38.247 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_auth_url": "", 2020-03-09T00:24:38.247 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_custom_header": "", 2020-03-09T00:24:38.247 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_enforce_content_length": "false", 2020-03-09T00:24:38.247 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_need_stats": "true", 2020-03-09T00:24:38.247 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_tenant_name": "", 2020-03-09T00:24:38.247 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_token_expiration": "86400", 2020-03-09T00:24:38.247 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_url": "", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_url_prefix": "swift", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_swift_versioning_enabled": "false", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_lease_period": "120", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_log_trim_interval": "1200", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_log_trim_max_buckets": "16", 2020-03-09T00:24:38.248 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2020-03-09T00:24:38.249 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2020-03-09T00:24:38.249 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_trace_history_size": "4096", 2020-03-09T00:24:38.249 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_trace_per_node_log_size": "32", 2020-03-09T00:24:38.249 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2020-03-09T00:24:38.249 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_thread_pool_size": "512", 2020-03-09T00:24:38.249 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_torrent_comment": "", 2020-03-09T00:24:38.249 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_torrent_createby": "", 2020-03-09T00:24:38.250 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_torrent_encoding": "", 2020-03-09T00:24:38.250 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_torrent_flag": "false", 2020-03-09T00:24:38.250 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_torrent_origin": "", 2020-03-09T00:24:38.250 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_torrent_sha_unit": "524288", 2020-03-09T00:24:38.251 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_torrent_tracker": "", 2020-03-09T00:24:38.251 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_trust_forwarded_https": "false", 2020-03-09T00:24:38.251 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_usage_log_flush_threshold": "1024", 2020-03-09T00:24:38.251 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_usage_log_tick_interval": "30", 2020-03-09T00:24:38.251 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_usage_max_shards": "32", 2020-03-09T00:24:38.251 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_usage_max_user_shards": "1", 2020-03-09T00:24:38.252 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_use_opa_authz": "false", 2020-03-09T00:24:38.252 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_default_quota_max_objects": "-1", 2020-03-09T00:24:38.252 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_default_quota_max_size": "-1", 2020-03-09T00:24:38.252 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_max_buckets": "1000", 2020-03-09T00:24:38.253 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2020-03-09T00:24:38.253 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_quota_sync_idle_users": "false", 2020-03-09T00:24:38.253 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_quota_sync_interval": "86400", 2020-03-09T00:24:38.253 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_quota_sync_wait_time": "86400", 2020-03-09T00:24:38.254 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_user_unique_email": "true", 2020-03-09T00:24:38.254 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_verify_ssl": "true", 2020-03-09T00:24:38.254 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_website_routing_rules_max_num": "50", 2020-03-09T00:24:38.254 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_zone": "", 2020-03-09T00:24:38.254 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_zone_root_pool": ".rgw.root", 2020-03-09T00:24:38.254 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_zonegroup": "", 2020-03-09T00:24:38.255 INFO:teuthology.orchestra.run.smithi148.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2020-03-09T00:24:38.255 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_block_size": "4096", 2020-03-09T00:24:38.255 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_bloom_bits_per_key": "20", 2020-03-09T00:24:38.255 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2020-03-09T00:24:38.255 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "true", 2020-03-09T00:24:38.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_cache_row_ratio": "0.000000", 2020-03-09T00:24:38.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_cache_shard_bits": "4", 2020-03-09T00:24:38.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_cache_size": "536870912", 2020-03-09T00:24:38.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_cache_type": "binned_lru", 2020-03-09T00:24:38.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_collect_compaction_stats": "false", 2020-03-09T00:24:38.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_collect_extended_stats": "false", 2020-03-09T00:24:38.257 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_collect_memory_stats": "false", 2020-03-09T00:24:38.257 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_delete_range_threshold": "1048576", 2020-03-09T00:24:38.257 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_index_type": "binary_search", 2020-03-09T00:24:38.257 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_log_to_ceph_log": "true", 2020-03-09T00:24:38.257 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_metadata_block_size": "4096", 2020-03-09T00:24:38.257 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_partition_filters": "false", 2020-03-09T00:24:38.257 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_perf": "false", 2020-03-09T00:24:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2020-03-09T00:24:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: "rotating_keys_bootstrap_timeout": "30", 2020-03-09T00:24:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: "rotating_keys_renewal_timeout": "10", 2020-03-09T00:24:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: "run_dir": "/var/run/ceph", 2020-03-09T00:24:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: "setgroup": "", 2020-03-09T00:24:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: "setuser": "", 2020-03-09T00:24:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: "setuser_match_path": "", 2020-03-09T00:24:38.259 INFO:teuthology.orchestra.run.smithi148.stdout: "target_max_misplaced_ratio": "0.050000", 2020-03-09T00:24:38.259 INFO:teuthology.orchestra.run.smithi148.stdout: "thp": "false", 2020-03-09T00:24:38.259 INFO:teuthology.orchestra.run.smithi148.stdout: "threadpool_default_timeout": "60", 2020-03-09T00:24:38.259 INFO:teuthology.orchestra.run.smithi148.stdout: "threadpool_empty_queue_max_wait": "2", 2020-03-09T00:24:38.259 INFO:teuthology.orchestra.run.smithi148.stdout: "throttler_perf_counter": "true", 2020-03-09T00:24:38.259 INFO:teuthology.orchestra.run.smithi148.stdout: "timing": "false" 2020-03-09T00:24:38.259 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:38.263 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2020-03-09T00:24:38.386 INFO:teuthology.orchestra.run.smithi148.stdout:mon.b: {} 2020-03-09T00:24:38.396 INFO:teuthology.orchestra.run.smithi148.stdout:mon.c: {} 2020-03-09T00:24:38.414 INFO:teuthology.orchestra.run.smithi148.stdout:mon.a: {} 2020-03-09T00:24:38.414 INFO:teuthology.orchestra.run.smithi148.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-03-09T00:24:38.417 INFO:teuthology.orchestra.run.smithi148.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-03-09T00:24:38.417 INFO:teuthology.orchestra.run.smithi148.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-03-09T00:24:38.418 INFO:teuthology.orchestra.run.smithi148:> type ceph-objectstore-tool 2020-03-09T00:24:38.435 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-03-09T00:24:38.435 INFO:teuthology.orchestra.run.smithi104:> type ceph-objectstore-tool 2020-03-09T00:24:38.452 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-03-09T00:24:38.453 INFO:teuthology.orchestra.run.smithi139:> type ceph-objectstore-tool 2020-03-09T00:24:38.470 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-03-09T00:24:38.470 INFO:teuthology.orchestra.run.smithi050:> type ceph-objectstore-tool 2020-03-09T00:24:38.488 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-03-09T00:24:38.488 INFO:teuthology.run_tasks:Running task rados... 2020-03-09T00:24:38.501 INFO:tasks.rados:Beginning rados... 2020-03-09T00:24:38.502 DEBUG:teuthology.run_tasks:Unwinding manager rados 2020-03-09T00:24:38.513 INFO:tasks.rados:joining rados 2020-03-09T00:24:38.513 INFO:tasks.thrashosds.thrasher:starting do_thrash 2020-03-09T00:24:38.513 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:24:38.514 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:24:38.514 INFO:tasks.thrashosds.thrasher:inject_pause on 11 2020-03-09T00:24:38.514 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2020-03-09T00:24:38.514 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2020-03-09T00:24:38.514 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok config set filestore_inject_stall 3 2020-03-09T00:24:38.515 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2020-03-09T00:24:38.515 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2020-03-09T00:24:38.516 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:24:38.516 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2020-03-09T00:24:38.516 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:24:38.517 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2020-03-09T00:24:38.517 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-03-09T00:24:38.517 INFO:tasks.rados:clients are ['client.0'] 2020-03-09T00:24:38.518 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd erasure-code-profile set shecprofile c=2 name=shecprofile plugin=shec crush-failure-domain=osd k=4 m=3 2020-03-09T00:24:38.519 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:38.515+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:38.617 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:38.616+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:38.658 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:38.658 INFO:teuthology.orchestra.run.smithi104.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2020-03-09T00:24:38.659 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:38.660 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:38.660 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:38.661 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:38.661 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:24:38.661 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:38.669 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:24:38.682 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:38.703 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:38.719 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:38.717+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:38.725 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:38.744 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:38.765 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:38.785 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:38.785 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:38.785 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:24:38.785 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:24:38.785 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:38.787 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:38.793 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:24:38.807 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:38.819 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:38.817+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:38.827 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:38.848 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:38.869 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:38.888 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:38.908 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:38.909 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:38.910 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:38.918 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:24:38.920 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:38.917+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:38.929 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:38.930 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:38.927+0000 7f329f7fe700 0 -- 172.21.15.148:0/3370595368 >> [v2:172.21.15.139:6818/26981,v1:172.21.15.139:6821/26981] conn(0x7f328002f250 msgr2=0x7f3280031640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:38.950 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:38.970 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:38.971 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.971 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.972 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.972 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.972 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.972 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.972 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.972 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.972 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.973 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.973 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.973 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.973 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.973 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.973 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:24:38.973 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:24:39.021 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:39.020+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:39.047 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:39.048 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:39.056 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:24:39.122 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:39.120+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:39.213 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:39.213 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:39.221 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:24:39.223 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:39.222+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:39.324 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:39.323+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:39.336 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:39.336 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:39.345 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:24:39.425 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:39.424+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:39.481 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:39.481 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:39.490 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:24:39.534 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:39.532+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:39.614 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:39.614 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:39.623 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:24:39.627 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:39.625+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:39.728 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:39.726+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:39.788 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:39.788 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:load: shec 2020-03-09T00:24:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is set 2020-03-09T00:24:39.797 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:24:39.802 INFO:tasks.rados:starting run 0 out of 1 2020-03-09T00:24:39.802 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2020-03-09T00:24:39.803 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool create unique_pool_0 16 16 erasure shecprofile 2020-03-09T00:24:39.828 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:39.826+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:39.921 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:39.922 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:39.929 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:39.927+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:39.930 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:24:40.030 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:40.027+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:40.050 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:40.050 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:40.059 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:24:40.130 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:40.129+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:40.216 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:40.216 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:40.224 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:24:40.231 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:40.228+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:40.332 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:40.330+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:40.351 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:40.351 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:40.362 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:24:40.445 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:40.443+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:40.516 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:40.517 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:40.525 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:24:40.566 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:40.564+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:40.634 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:40.633+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:40.679 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:40.680 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:40.688 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:24:40.735 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:40.734+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:40.816 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:40.816 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:40.825 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:24:40.836 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:40.834+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:40.937 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:40.936+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:40.943 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:40.943 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:40.951 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:24:40.981 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:24:41.038 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:41.036+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:41.066 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:41.067 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:41.075 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:24:41.116 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:41.136 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:41.138 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:41.138+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:41.157 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:41.177 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:41.197 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:41.212 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:41.212 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:24:41.213 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:41.218 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:41.222 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:24:41.238 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:41.239 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:41.236+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:41.258 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:41.277 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:41.298 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:41.318 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:41.338 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:41.340 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:41.338+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:24:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:41.346 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:41.347 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:24:41.347 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:24:41.347 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:41.355 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:24:41.357 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:41.377 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:41.378 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:41.376+0000 7fef0b7fe700 0 -- 172.21.15.148:0/125420473 >> [v2:172.21.15.139:6818/26981,v1:172.21.15.139:6821/26981] conn(0x7fef0c189a30 msgr2=0x7fef0c18be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:41.441 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:41.440+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:41.519 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:24:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:24:41.519 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2020-03-09T00:24:41.519 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:41.528 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:24:41.571 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:41.569+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:41.617 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:41.638 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:41.640 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.640 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.640 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.640 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.640 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.640 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:24:41.644 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:41.641+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:41.661 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:41.662 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:41.662 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:24:41.662 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:41.670 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:24:41.744 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:41.743+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:41.791 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'unique_pool_0' created 2020-03-09T00:24:41.803 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-03-09T00:24:41.806 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2020-03-09T00:24:41.835 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:41.835 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:41.835 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:24:41.835 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:24:41.835 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:41.846 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:24:41.846 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:41.843+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:41.946 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:41.943+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:42.015 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:42.016 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:24:42.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:24:42.016 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:24:42.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.016 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.4118.0:2 1.0 e26/24 1:e8e3cb91:::INTEL_SSDPED1D480GA_PHMB7513001F480DGN:head v 26'1, mlcod=0'0)", 2020-03-09T00:24:42.016 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:24.664052+0000", 2020-03-09T00:24:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 17.350579575000001, 2020-03-09T00:24:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076192800000000004, 2020-03-09T00:24:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:42.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664052+0000", 2020-03-09T00:24:42.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:42.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:42.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664052+0000", 2020-03-09T00:24:42.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7836999999999994e-05 2020-03-09T00:24:42.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:42.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664120+0000", 2020-03-09T00:24:42.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7559999999999994e-06 2020-03-09T00:24:42.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664130+0000", 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1759999999999999e-05 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664141+0000", 2020-03-09T00:24:42.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0610000000000002e-06 2020-03-09T00:24:42.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:42.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664150+0000", 2020-03-09T00:24:42.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3239999999999998e-05 2020-03-09T00:24:42.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:42.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664214+0000", 2020-03-09T00:24:42.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0395999999999995e-05 2020-03-09T00:24:42.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:42.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664284+0000", 2020-03-09T00:24:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00046297999999999999 2020-03-09T00:24:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:24:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664747+0000", 2020-03-09T00:24:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6897999999999995e-05 2020-03-09T00:24:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:42.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.664814+0000", 2020-03-09T00:24:42.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076192800000000004 2020-03-09T00:24:42.024 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:42.024 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:42.024 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:42.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.025 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.4118.0:4 1.0 e26/24 1:a3830038:::INTEL_SSDPEDMD400G4_PHFT620500N6400BGN:head v 26'2, mlcod=0'0)", 2020-03-09T00:24:42.025 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:24.746668+0000", 2020-03-09T00:24:42.025 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 17.267963145, 2020-03-09T00:24:42.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00082831300000000001, 2020-03-09T00:24:42.026 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:42.026 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:42.026 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:42.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:42.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746668+0000", 2020-03-09T00:24:42.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746668+0000", 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018892599999999999 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:42.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746857+0000", 2020-03-09T00:24:42.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.343e-05 2020-03-09T00:24:42.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:42.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746871+0000", 2020-03-09T00:24:42.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0729e-05 2020-03-09T00:24:42.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:42.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746882+0000", 2020-03-09T00:24:42.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9480000000000002e-06 2020-03-09T00:24:42.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:42.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746888+0000", 2020-03-09T00:24:42.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1073000000000002e-05 2020-03-09T00:24:42.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:42.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746960+0000", 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8558000000000001e-05 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.746978+0000", 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00045518499999999999 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:24:42.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.747433+0000", 2020-03-09T00:24:42.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3464000000000001e-05 2020-03-09T00:24:42.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:42.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.747497+0000", 2020-03-09T00:24:42.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00082831300000000001 2020-03-09T00:24:42.033 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:42.033 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:42.033 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:42.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.034 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_repop(client.4118.0:6 1.0 e26/24 1:15e1a584:::INTEL_SSDPEDMD400G4_CVFT6233005G400BGN:head v 26'3, mlcod=26'1)", 2020-03-09T00:24:42.034 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:24.844405+0000", 2020-03-09T00:24:42.034 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 17.170226991, 2020-03-09T00:24:42.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001005417, 2020-03-09T00:24:42.034 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:42.034 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844405+0000", 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:42.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844405+0000", 2020-03-09T00:24:42.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1872000000000007e-05 2020-03-09T00:24:42.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:42.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844466+0000", 2020-03-09T00:24:42.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1584e-05 2020-03-09T00:24:42.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:42.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844478+0000", 2020-03-09T00:24:42.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4409e-05 2020-03-09T00:24:42.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:42.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844492+0000", 2020-03-09T00:24:42.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5459999999999997e-06 2020-03-09T00:24:42.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:42.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844502+0000", 2020-03-09T00:24:42.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3855000000000004e-05 2020-03-09T00:24:42.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:42.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844576+0000", 2020-03-09T00:24:42.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.2912999999999999e-05 2020-03-09T00:24:42.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:42.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.844599+0000", 2020-03-09T00:24:42.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000748684 2020-03-09T00:24:42.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:24:42.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.845347+0000", 2020-03-09T00:24:42.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2553999999999998e-05 2020-03-09T00:24:42.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:42.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:42.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:42.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:24.845410+0000", 2020-03-09T00:24:42.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001005417 2020-03-09T00:24:42.042 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:42.042 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:42.042 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:42.042 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:42.042 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:42.042 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:42.043 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:24:42.084 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:42.082+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:42.147 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:42.146+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:42.169 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:42.169 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:42.169 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:24:42.169 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:42.178 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:24:42.249 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:42.247+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:42.299 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:24:42.299 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:24:42.299 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:42.307 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:24:42.351 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:42.350+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:42.424 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:42.424 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:24:42.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:24:42.425 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [] 2020-03-09T00:24:42.425 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:42.432 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:24:42.476 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:42.475+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:42.551 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:42.549+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:42.560 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:42.560 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:42.560 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:24:42.560 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:42.575 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:24:42.658 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:42.657+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:42.731 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:42.731 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:42.731 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:24:42.731 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:24:42.731 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:42.739 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:24:42.753 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:42.751+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is set 2020-03-09T00:24:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:enabled application 'rados' on pool 'unique_pool_0' 2020-03-09T00:24:42.853 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:42.852+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:42.903 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:24:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:24:42.903 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [] 2020-03-09T00:24:42.903 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:42.911 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:24:42.954 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:42.953+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:43.049 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:43.049 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:43.049 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:24:43.049 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:43.055 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:43.054+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:43.060 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:24:43.157 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:43.155+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:43.186 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:43.186 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:43.186 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:24:43.186 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:24:43.186 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:43.195 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:24:43.258 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:43.256+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:43.319 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:43.320 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:24:43.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:24:43.320 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [] 2020-03-09T00:24:43.320 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:43.328 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:24:43.359 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:43.358+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:43.457 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:24:43.457 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:43.461 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:43.458+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:43.466 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:24:43.561 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:43.560+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:43.622 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:24:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:24:43.622 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:43.630 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:24:43.650 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:24:43.662 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:43.660+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:43.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:24:43.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:24:43.672 INFO:tasks.thrashosds.thrasher:inject_pause on 8 2020-03-09T00:24:43.672 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2020-03-09T00:24:43.672 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2020-03-09T00:24:43.672 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok config set filestore_inject_stall 3 2020-03-09T00:24:43.763 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:43.761+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:43.764 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:43.764 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:24:43.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:24:43.765 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [] 2020-03-09T00:24:43.765 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:43.774 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:24:43.808 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:43.809 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2020-03-09T00:24:43.809 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:43.813 INFO:teuthology.orchestra.run.smithi104:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op snap_remove 50 --op snap_create 50 --op rollback 50 --op setattr 25 --op read 100 --op copy_from 50 --op write 0 --op rmattr 25 --op append 100 --op delete 50 --pool unique_pool_0 2020-03-09T00:24:43.816 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:43.837 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:43.841 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight snap_remove -> 50 2020-03-09T00:24:43.842 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight snap_create -> 50 2020-03-09T00:24:43.842 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight rollback -> 50 2020-03-09T00:24:43.842 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight setattr -> 25 2020-03-09T00:24:43.842 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight read -> 100 2020-03-09T00:24:43.842 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight copy_from -> 50 2020-03-09T00:24:43.842 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight rmattr -> 25 2020-03-09T00:24:43.842 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight append -> 100 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout:adding op weight delete -> 50 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout:ceph version 15.1.0-1904-g9eb9490 (9eb9490ed0531fd33292066429bf9db55b9c1d3b) octopus (rc) 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout:Configuration: 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout: Number of operations: 400 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout: Number of objects: 50 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout: Max in flight operations: 16 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout: Object size (in bytes): 4000000 2020-03-09T00:24:43.843 INFO:tasks.rados.rados.0.smithi104.stdout: Write stride min: 400000 2020-03-09T00:24:43.844 INFO:tasks.rados.rados.0.smithi104.stdout: Write stride max: 800000 2020-03-09T00:24:43.859 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:43.864 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:43.862+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:43.879 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:43.899 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:43.903 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:43.903 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:24:43.903 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:43.911 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:24:43.918 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:43.939 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:43.959 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:43.965 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:43.963+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:43.979 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:44.001 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:44.019 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:44.039 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:44.059 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:44.065 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:44.062+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:44.066 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:44.066 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:44.075 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:24:44.080 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:44.080 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:44.078+0000 7fdada59c700 0 -- 172.21.15.148:0/996379253 >> [v2:172.21.15.139:6818/26981,v1:172.21.15.139:6821/26981] conn(0x7fdadc187600 msgr2=0x7fdadc1899f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:44.166 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:44.164+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:44.189 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:44.190 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:44.199 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-03-09T00:24:44.267 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:44.265+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:44.302 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:44.322 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:24:44.325 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:44.325 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:44.333 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-03-09T00:24:44.373 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:44.372+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:44.448 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:44.449 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:44.457 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-03-09T00:24:44.468 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:44.466+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:44.569 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:44.567+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:44.573 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:44.573 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:44.582 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:24:44.671 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:44.669+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:44.718 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:44.718 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:44.727 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:24:44.771 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:44.769+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:44.798 INFO:tasks.rados.rados.0.smithi104.stdout:1: write initial oid 1 2020-03-09T00:24:44.798 INFO:tasks.rados.rados.0.smithi104.stdout:1: seq_num 0 ranges {0=32768} 2020-03-09T00:24:44.798 INFO:tasks.rados.rados.0.smithi104.stdout:1: writing smithi10428818-1 from 0 to 32768 tid 1 2020-03-09T00:24:44.799 INFO:tasks.rados.rados.0.smithi104.stdout:2: write initial oid 2 2020-03-09T00:24:44.799 INFO:tasks.rados.rados.0.smithi104.stdout:2: seq_num 1 ranges {0=32768} 2020-03-09T00:24:44.799 INFO:tasks.rados.rados.0.smithi104.stdout:2: writing smithi10428818-2 from 0 to 32768 tid 1 2020-03-09T00:24:44.799 INFO:tasks.rados.rados.0.smithi104.stdout:3: write initial oid 3 2020-03-09T00:24:44.799 INFO:tasks.rados.rados.0.smithi104.stdout:3: seq_num 2 ranges {0=49152} 2020-03-09T00:24:44.800 INFO:tasks.rados.rados.0.smithi104.stdout:3: writing smithi10428818-3 from 0 to 49152 tid 1 2020-03-09T00:24:44.800 INFO:tasks.rados.rados.0.smithi104.stdout:4: write initial oid 4 2020-03-09T00:24:44.800 INFO:tasks.rados.rados.0.smithi104.stdout:4: seq_num 3 ranges {0=65536} 2020-03-09T00:24:44.801 INFO:tasks.rados.rados.0.smithi104.stdout:4: writing smithi10428818-4 from 0 to 65536 tid 1 2020-03-09T00:24:44.801 INFO:tasks.rados.rados.0.smithi104.stdout:5: write initial oid 5 2020-03-09T00:24:44.801 INFO:tasks.rados.rados.0.smithi104.stdout:5: seq_num 4 ranges {0=81920} 2020-03-09T00:24:44.801 INFO:tasks.rados.rados.0.smithi104.stdout:5: writing smithi10428818-5 from 0 to 81920 tid 1 2020-03-09T00:24:44.802 INFO:tasks.rados.rados.0.smithi104.stdout:6: write initial oid 6 2020-03-09T00:24:44.802 INFO:tasks.rados.rados.0.smithi104.stdout:6: seq_num 5 ranges {0=98304} 2020-03-09T00:24:44.802 INFO:tasks.rados.rados.0.smithi104.stdout:6: writing smithi10428818-6 from 0 to 98304 tid 1 2020-03-09T00:24:44.802 INFO:tasks.rados.rados.0.smithi104.stdout:7: write initial oid 7 2020-03-09T00:24:44.802 INFO:tasks.rados.rados.0.smithi104.stdout:7: seq_num 6 ranges {0=114688} 2020-03-09T00:24:44.802 INFO:tasks.rados.rados.0.smithi104.stdout:7: writing smithi10428818-7 from 0 to 114688 tid 1 2020-03-09T00:24:44.802 INFO:tasks.rados.rados.0.smithi104.stdout:8: write initial oid 8 2020-03-09T00:24:44.803 INFO:tasks.rados.rados.0.smithi104.stdout:8: seq_num 7 ranges {0=131072} 2020-03-09T00:24:44.803 INFO:tasks.rados.rados.0.smithi104.stdout:8: writing smithi10428818-8 from 0 to 131072 tid 1 2020-03-09T00:24:44.803 INFO:tasks.rados.rados.0.smithi104.stdout:9: write initial oid 9 2020-03-09T00:24:44.804 INFO:tasks.rados.rados.0.smithi104.stdout:9: seq_num 8 ranges {0=147456} 2020-03-09T00:24:44.804 INFO:tasks.rados.rados.0.smithi104.stdout:9: writing smithi10428818-9 from 0 to 147456 tid 1 2020-03-09T00:24:44.804 INFO:tasks.rados.rados.0.smithi104.stdout:10: write initial oid 10 2020-03-09T00:24:44.804 INFO:tasks.rados.rados.0.smithi104.stdout:10: seq_num 9 ranges {0=163840} 2020-03-09T00:24:44.805 INFO:tasks.rados.rados.0.smithi104.stdout:10: writing smithi10428818-10 from 0 to 163840 tid 1 2020-03-09T00:24:44.805 INFO:tasks.rados.rados.0.smithi104.stdout:11: write initial oid 11 2020-03-09T00:24:44.805 INFO:tasks.rados.rados.0.smithi104.stdout:11: seq_num 10 ranges {0=180224} 2020-03-09T00:24:44.806 INFO:tasks.rados.rados.0.smithi104.stdout:11: writing smithi10428818-11 from 0 to 180224 tid 1 2020-03-09T00:24:44.806 INFO:tasks.rados.rados.0.smithi104.stdout:12: write initial oid 12 2020-03-09T00:24:44.806 INFO:tasks.rados.rados.0.smithi104.stdout:12: seq_num 11 ranges {0=196608} 2020-03-09T00:24:44.807 INFO:tasks.rados.rados.0.smithi104.stdout:12: writing smithi10428818-12 from 0 to 196608 tid 1 2020-03-09T00:24:44.807 INFO:tasks.rados.rados.0.smithi104.stdout:13: write initial oid 13 2020-03-09T00:24:44.807 INFO:tasks.rados.rados.0.smithi104.stdout:13: seq_num 12 ranges {0=212992} 2020-03-09T00:24:44.808 INFO:tasks.rados.rados.0.smithi104.stdout:13: writing smithi10428818-13 from 0 to 212992 tid 1 2020-03-09T00:24:44.809 INFO:tasks.rados.rados.0.smithi104.stdout:14: write initial oid 14 2020-03-09T00:24:44.809 INFO:tasks.rados.rados.0.smithi104.stdout:14: seq_num 13 ranges {0=229376} 2020-03-09T00:24:44.810 INFO:tasks.rados.rados.0.smithi104.stdout:14: writing smithi10428818-14 from 0 to 229376 tid 1 2020-03-09T00:24:44.810 INFO:tasks.rados.rados.0.smithi104.stdout:15: write initial oid 15 2020-03-09T00:24:44.810 INFO:tasks.rados.rados.0.smithi104.stdout:15: seq_num 14 ranges {0=245760} 2020-03-09T00:24:44.812 INFO:tasks.rados.rados.0.smithi104.stdout:15: writing smithi10428818-15 from 0 to 245760 tid 1 2020-03-09T00:24:44.812 INFO:tasks.rados.rados.0.smithi104.stdout:16: write initial oid 16 2020-03-09T00:24:44.812 INFO:tasks.rados.rados.0.smithi104.stdout:16: seq_num 15 ranges {0=262144} 2020-03-09T00:24:44.813 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-03-09T00:24:44.814 INFO:tasks.rados.rados.0.smithi104.stdout:16: writing smithi10428818-16 from 0 to 262144 tid 1 2020-03-09T00:24:44.815 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.815 INFO:tasks.rados.rados.0.smithi104.stdout:1: finishing write tid 1 to smithi10428818-1 2020-03-09T00:24:44.817 INFO:tasks.rados.rados.0.smithi104.stdout:3: finishing write tid 1 to smithi10428818-3 2020-03-09T00:24:44.817 INFO:tasks.rados.rados.0.smithi104.stdout:2: finishing write tid 1 to smithi10428818-2 2020-03-09T00:24:44.817 INFO:tasks.rados.rados.0.smithi104.stdout:7: finishing write tid 1 to smithi10428818-7 2020-03-09T00:24:44.817 INFO:tasks.rados.rados.0.smithi104.stdout:4: finishing write tid 1 to smithi10428818-4 2020-03-09T00:24:44.817 INFO:tasks.rados.rados.0.smithi104.stdout:5: finishing write tid 1 to smithi10428818-5 2020-03-09T00:24:44.817 INFO:tasks.rados.rados.0.smithi104.stdout:1: finishing write tid 3 to smithi10428818-1 2020-03-09T00:24:44.817 INFO:tasks.rados.rados.0.smithi104.stdout:6: finishing write tid 1 to smithi10428818-6 2020-03-09T00:24:44.818 INFO:tasks.rados.rados.0.smithi104.stdout:10: finishing write tid 1 to smithi10428818-10 2020-03-09T00:24:44.819 INFO:tasks.rados.rados.0.smithi104.stdout:8: finishing write tid 1 to smithi10428818-8 2020-03-09T00:24:44.819 INFO:tasks.rados.rados.0.smithi104.stdout:3: finishing write tid 3 to smithi10428818-3 2020-03-09T00:24:44.819 INFO:tasks.rados.rados.0.smithi104.stdout:4: finishing write tid 3 to smithi10428818-4 2020-03-09T00:24:44.822 INFO:tasks.rados.rados.0.smithi104.stdout:9: finishing write tid 1 to smithi10428818-9 2020-03-09T00:24:44.822 INFO:tasks.rados.rados.0.smithi104.stdout:1: finishing write tid 4 to smithi10428818-1 2020-03-09T00:24:44.822 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-03-09T00:24:44.822 INFO:tasks.rados.rados.0.smithi104.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2020-03-09T00:24:44.822 INFO:tasks.rados.rados.0.smithi104.stdout:1: done (15 left) 2020-03-09T00:24:44.823 INFO:tasks.rados.rados.0.smithi104.stdout:17: write initial oid 17 2020-03-09T00:24:44.823 INFO:tasks.rados.rados.0.smithi104.stdout:17: seq_num 16 ranges {0=278528} 2020-03-09T00:24:44.824 INFO:tasks.rados.rados.0.smithi104.stdout:17: writing smithi10428818-17 from 0 to 278528 tid 1 2020-03-09T00:24:44.824 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.824 INFO:tasks.rados.rados.0.smithi104.stdout:2: finishing write tid 3 to smithi10428818-2 2020-03-09T00:24:44.825 INFO:tasks.rados.rados.0.smithi104.stdout:3: finishing write tid 4 to smithi10428818-3 2020-03-09T00:24:44.825 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-03-09T00:24:44.825 INFO:tasks.rados.rados.0.smithi104.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2020-03-09T00:24:44.825 INFO:tasks.rados.rados.0.smithi104.stdout:3: done (15 left) 2020-03-09T00:24:44.825 INFO:tasks.rados.rados.0.smithi104.stdout:18: write initial oid 18 2020-03-09T00:24:44.825 INFO:tasks.rados.rados.0.smithi104.stdout:18: seq_num 17 ranges {0=294912} 2020-03-09T00:24:44.827 INFO:tasks.rados.rados.0.smithi104.stdout:18: writing smithi10428818-18 from 0 to 294912 tid 1 2020-03-09T00:24:44.828 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.828 INFO:tasks.rados.rados.0.smithi104.stdout:13: finishing write tid 1 to smithi10428818-13 2020-03-09T00:24:44.828 INFO:tasks.rados.rados.0.smithi104.stdout:4: finishing write tid 4 to smithi10428818-4 2020-03-09T00:24:44.828 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 4 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-03-09T00:24:44.828 INFO:tasks.rados.rados.0.smithi104.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2020-03-09T00:24:44.828 INFO:tasks.rados.rados.0.smithi104.stdout:6: finishing write tid 3 to smithi10428818-6 2020-03-09T00:24:44.829 INFO:tasks.rados.rados.0.smithi104.stdout:7: finishing write tid 3 to smithi10428818-7 2020-03-09T00:24:44.829 INFO:tasks.rados.rados.0.smithi104.stdout:4: done (15 left) 2020-03-09T00:24:44.829 INFO:tasks.rados.rados.0.smithi104.stdout:19: write initial oid 19 2020-03-09T00:24:44.829 INFO:tasks.rados.rados.0.smithi104.stdout:19: seq_num 18 ranges {0=311296} 2020-03-09T00:24:44.830 INFO:tasks.rados.rados.0.smithi104.stdout:19: writing smithi10428818-19 from 0 to 311296 tid 1 2020-03-09T00:24:44.830 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.830 INFO:tasks.rados.rados.0.smithi104.stdout:2: finishing write tid 4 to smithi10428818-2 2020-03-09T00:24:44.831 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-03-09T00:24:44.831 INFO:tasks.rados.rados.0.smithi104.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2020-03-09T00:24:44.831 INFO:tasks.rados.rados.0.smithi104.stdout:8: finishing write tid 3 to smithi10428818-8 2020-03-09T00:24:44.831 INFO:tasks.rados.rados.0.smithi104.stdout:10: finishing write tid 3 to smithi10428818-10 2020-03-09T00:24:44.831 INFO:tasks.rados.rados.0.smithi104.stdout:5: finishing write tid 3 to smithi10428818-5 2020-03-09T00:24:44.831 INFO:tasks.rados.rados.0.smithi104.stdout:12: finishing write tid 1 to smithi10428818-12 2020-03-09T00:24:44.831 INFO:tasks.rados.rados.0.smithi104.stdout:2: done (15 left) 2020-03-09T00:24:44.832 INFO:tasks.rados.rados.0.smithi104.stdout:20: write initial oid 20 2020-03-09T00:24:44.832 INFO:tasks.rados.rados.0.smithi104.stdout:20: seq_num 19 ranges {0=327680} 2020-03-09T00:24:44.833 INFO:tasks.rados.rados.0.smithi104.stdout:20: writing smithi10428818-20 from 0 to 327680 tid 1 2020-03-09T00:24:44.833 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.833 INFO:tasks.rados.rados.0.smithi104.stdout:7: finishing write tid 4 to smithi10428818-7 2020-03-09T00:24:44.833 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-03-09T00:24:44.834 INFO:tasks.rados.rados.0.smithi104.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2020-03-09T00:24:44.834 INFO:tasks.rados.rados.0.smithi104.stdout:13: finishing write tid 3 to smithi10428818-13 2020-03-09T00:24:44.834 INFO:tasks.rados.rados.0.smithi104.stdout:7: done (15 left) 2020-03-09T00:24:44.834 INFO:tasks.rados.rados.0.smithi104.stdout:21: write initial oid 21 2020-03-09T00:24:44.834 INFO:tasks.rados.rados.0.smithi104.stdout:21: seq_num 20 ranges {0=344064} 2020-03-09T00:24:44.837 INFO:tasks.rados.rados.0.smithi104.stdout:21: writing smithi10428818-21 from 0 to 344064 tid 1 2020-03-09T00:24:44.837 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.837 INFO:tasks.rados.rados.0.smithi104.stdout:9: finishing write tid 3 to smithi10428818-9 2020-03-09T00:24:44.837 INFO:tasks.rados.rados.0.smithi104.stdout:10: finishing write tid 4 to smithi10428818-10 2020-03-09T00:24:44.837 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-03-09T00:24:44.837 INFO:tasks.rados.rados.0.smithi104.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:11: finishing write tid 1 to smithi10428818-11 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:15: finishing write tid 1 to smithi10428818-15 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:17: finishing write tid 1 to smithi10428818-17 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:8: finishing write tid 4 to smithi10428818-8 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:8: done (15 left) 2020-03-09T00:24:44.838 INFO:tasks.rados.rados.0.smithi104.stdout:10: done (14 left) 2020-03-09T00:24:44.839 INFO:tasks.rados.rados.0.smithi104.stdout:22: write initial oid 22 2020-03-09T00:24:44.839 INFO:tasks.rados.rados.0.smithi104.stdout:22: seq_num 21 ranges {0=360448} 2020-03-09T00:24:44.839 INFO:tasks.rados.rados.0.smithi104.stdout:22: writing smithi10428818-22 from 0 to 360448 tid 1 2020-03-09T00:24:44.839 INFO:tasks.rados.rados.0.smithi104.stdout:23: write initial oid 23 2020-03-09T00:24:44.840 INFO:tasks.rados.rados.0.smithi104.stdout:23: seq_num 22 ranges {0=376832} 2020-03-09T00:24:44.841 INFO:tasks.rados.rados.0.smithi104.stdout:23: writing smithi10428818-23 from 0 to 376832 tid 1 2020-03-09T00:24:44.841 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.841 INFO:tasks.rados.rados.0.smithi104.stdout:16: finishing write tid 1 to smithi10428818-16 2020-03-09T00:24:44.841 INFO:tasks.rados.rados.0.smithi104.stdout:14: finishing write tid 1 to smithi10428818-14 2020-03-09T00:24:44.841 INFO:tasks.rados.rados.0.smithi104.stdout:17: finishing write tid 3 to smithi10428818-17 2020-03-09T00:24:44.841 INFO:tasks.rados.rados.0.smithi104.stdout:9: finishing write tid 4 to smithi10428818-9 2020-03-09T00:24:44.842 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-03-09T00:24:44.842 INFO:tasks.rados.rados.0.smithi104.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2020-03-09T00:24:44.842 INFO:tasks.rados.rados.0.smithi104.stdout:11: finishing write tid 3 to smithi10428818-11 2020-03-09T00:24:44.842 INFO:tasks.rados.rados.0.smithi104.stdout:15: finishing write tid 3 to smithi10428818-15 2020-03-09T00:24:44.842 INFO:tasks.rados.rados.0.smithi104.stdout:9: done (15 left) 2020-03-09T00:24:44.842 INFO:tasks.rados.rados.0.smithi104.stdout:24: write initial oid 24 2020-03-09T00:24:44.842 INFO:tasks.rados.rados.0.smithi104.stdout:24: seq_num 23 ranges {0=393216} 2020-03-09T00:24:44.844 INFO:tasks.rados.rados.0.smithi104.stdout:24: writing smithi10428818-24 from 0 to 393216 tid 1 2020-03-09T00:24:44.844 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.844 INFO:tasks.rados.rados.0.smithi104.stdout:5: finishing write tid 4 to smithi10428818-5 2020-03-09T00:24:44.844 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-03-09T00:24:44.844 INFO:tasks.rados.rados.0.smithi104.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2020-03-09T00:24:44.844 INFO:tasks.rados.rados.0.smithi104.stdout:12: finishing write tid 3 to smithi10428818-12 2020-03-09T00:24:44.845 INFO:tasks.rados.rados.0.smithi104.stdout:13: finishing write tid 4 to smithi10428818-13 2020-03-09T00:24:44.845 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-03-09T00:24:44.845 INFO:tasks.rados.rados.0.smithi104.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2020-03-09T00:24:44.845 INFO:tasks.rados.rados.0.smithi104.stdout:5: done (15 left) 2020-03-09T00:24:44.845 INFO:tasks.rados.rados.0.smithi104.stdout:13: done (14 left) 2020-03-09T00:24:44.845 INFO:tasks.rados.rados.0.smithi104.stdout:25: write initial oid 25 2020-03-09T00:24:44.845 INFO:tasks.rados.rados.0.smithi104.stdout:25: seq_num 24 ranges {0=409600} 2020-03-09T00:24:44.846 INFO:tasks.rados.rados.0.smithi104.stdout:25: writing smithi10428818-25 from 0 to 409600 tid 1 2020-03-09T00:24:44.846 INFO:tasks.rados.rados.0.smithi104.stdout:26: write initial oid 26 2020-03-09T00:24:44.847 INFO:tasks.rados.rados.0.smithi104.stdout:26: seq_num 25 ranges {0=425984} 2020-03-09T00:24:44.858 INFO:tasks.rados.rados.0.smithi104.stdout:26: writing smithi10428818-26 from 0 to 425984 tid 1 2020-03-09T00:24:44.858 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.858 INFO:tasks.rados.rados.0.smithi104.stdout:6: finishing write tid 4 to smithi10428818-6 2020-03-09T00:24:44.858 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:11: finishing write tid 4 to smithi10428818-11 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:17: finishing write tid 4 to smithi10428818-17 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2020-03-09T00:24:44.859 INFO:tasks.rados.rados.0.smithi104.stdout:20: finishing write tid 1 to smithi10428818-20 2020-03-09T00:24:44.860 INFO:tasks.rados.rados.0.smithi104.stdout:12: finishing write tid 4 to smithi10428818-12 2020-03-09T00:24:44.860 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-03-09T00:24:44.860 INFO:tasks.rados.rados.0.smithi104.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:24:44.860 INFO:tasks.rados.rados.0.smithi104.stdout:15: finishing write tid 4 to smithi10428818-15 2020-03-09T00:24:44.860 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-03-09T00:24:44.860 INFO:tasks.rados.rados.0.smithi104.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2020-03-09T00:24:44.860 INFO:tasks.rados.rados.0.smithi104.stdout:18: finishing write tid 1 to smithi10428818-18 2020-03-09T00:24:44.861 INFO:tasks.rados.rados.0.smithi104.stdout:14: finishing write tid 3 to smithi10428818-14 2020-03-09T00:24:44.861 INFO:tasks.rados.rados.0.smithi104.stdout:20: finishing write tid 3 to smithi10428818-20 2020-03-09T00:24:44.861 INFO:tasks.rados.rados.0.smithi104.stdout:6: done (15 left) 2020-03-09T00:24:44.861 INFO:tasks.rados.rados.0.smithi104.stdout:11: done (14 left) 2020-03-09T00:24:44.861 INFO:tasks.rados.rados.0.smithi104.stdout:12: done (13 left) 2020-03-09T00:24:44.861 INFO:tasks.rados.rados.0.smithi104.stdout:15: done (12 left) 2020-03-09T00:24:44.861 INFO:tasks.rados.rados.0.smithi104.stdout:17: done (11 left) 2020-03-09T00:24:44.862 INFO:tasks.rados.rados.0.smithi104.stdout:27: write initial oid 27 2020-03-09T00:24:44.863 INFO:tasks.rados.rados.0.smithi104.stdout:27: seq_num 26 ranges {0=442368} 2020-03-09T00:24:44.863 INFO:tasks.rados.rados.0.smithi104.stdout:27: writing smithi10428818-27 from 0 to 442368 tid 1 2020-03-09T00:24:44.863 INFO:tasks.rados.rados.0.smithi104.stdout:28: write initial oid 28 2020-03-09T00:24:44.863 INFO:tasks.rados.rados.0.smithi104.stdout:28: seq_num 27 ranges {0=458752} 2020-03-09T00:24:44.863 INFO:tasks.rados.rados.0.smithi104.stdout:28: writing smithi10428818-28 from 0 to 458752 tid 1 2020-03-09T00:24:44.863 INFO:tasks.rados.rados.0.smithi104.stdout:23: finishing write tid 1 to smithi10428818-23 2020-03-09T00:24:44.863 INFO:tasks.rados.rados.0.smithi104.stdout:19: finishing write tid 1 to smithi10428818-19 2020-03-09T00:24:44.864 INFO:tasks.rados.rados.0.smithi104.stdout:16: finishing write tid 3 to smithi10428818-16 2020-03-09T00:24:44.864 INFO:tasks.rados.rados.0.smithi104.stdout:18: finishing write tid 3 to smithi10428818-18 2020-03-09T00:24:44.864 INFO:tasks.rados.rados.0.smithi104.stdout:21: finishing write tid 1 to smithi10428818-21 2020-03-09T00:24:44.864 INFO:tasks.rados.rados.0.smithi104.stdout:22: finishing write tid 1 to smithi10428818-22 2020-03-09T00:24:44.864 INFO:tasks.rados.rados.0.smithi104.stdout:29: write initial oid 29 2020-03-09T00:24:44.864 INFO:tasks.rados.rados.0.smithi104.stdout:29: seq_num 28 ranges {0=475136} 2020-03-09T00:24:44.864 INFO:tasks.rados.rados.0.smithi104.stdout:29: writing smithi10428818-29 from 0 to 475136 tid 1 2020-03-09T00:24:44.865 INFO:tasks.rados.rados.0.smithi104.stdout:30: write initial oid 30 2020-03-09T00:24:44.865 INFO:tasks.rados.rados.0.smithi104.stdout:30: seq_num 29 ranges {0=491520} 2020-03-09T00:24:44.866 INFO:tasks.rados.rados.0.smithi104.stdout:30: writing smithi10428818-30 from 0 to 491520 tid 1 2020-03-09T00:24:44.866 INFO:tasks.rados.rados.0.smithi104.stdout:31: write initial oid 31 2020-03-09T00:24:44.867 INFO:tasks.rados.rados.0.smithi104.stdout:31: seq_num 30 ranges {0=507904} 2020-03-09T00:24:44.867 INFO:tasks.rados.rados.0.smithi104.stdout:31: writing smithi10428818-31 from 0 to 507904 tid 1 2020-03-09T00:24:44.867 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.867 INFO:tasks.rados.rados.0.smithi104.stdout:20: finishing write tid 4 to smithi10428818-20 2020-03-09T00:24:44.867 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 20 v 4 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-03-09T00:24:44.867 INFO:tasks.rados.rados.0.smithi104.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2020-03-09T00:24:44.868 INFO:tasks.rados.rados.0.smithi104.stdout:20: done (15 left) 2020-03-09T00:24:44.868 INFO:tasks.rados.rados.0.smithi104.stdout:32: write initial oid 32 2020-03-09T00:24:44.868 INFO:tasks.rados.rados.0.smithi104.stdout:32: seq_num 31 ranges {0=524288} 2020-03-09T00:24:44.870 INFO:tasks.rados.rados.0.smithi104.stdout:32: writing smithi10428818-32 from 0 to 524288 tid 1 2020-03-09T00:24:44.870 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.870 INFO:tasks.rados.rados.0.smithi104.stdout:24: finishing write tid 1 to smithi10428818-24 2020-03-09T00:24:44.870 INFO:tasks.rados.rados.0.smithi104.stdout:19: finishing write tid 3 to smithi10428818-19 2020-03-09T00:24:44.871 INFO:tasks.rados.rados.0.smithi104.stdout:14: finishing write tid 4 to smithi10428818-14 2020-03-09T00:24:44.871 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-03-09T00:24:44.871 INFO:tasks.rados.rados.0.smithi104.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2020-03-09T00:24:44.871 INFO:tasks.rados.rados.0.smithi104.stdout:18: finishing write tid 4 to smithi10428818-18 2020-03-09T00:24:44.871 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 18 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-03-09T00:24:44.871 INFO:tasks.rados.rados.0.smithi104.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2020-03-09T00:24:44.871 INFO:tasks.rados.rados.0.smithi104.stdout:26: finishing write tid 1 to smithi10428818-26 2020-03-09T00:24:44.872 INFO:tasks.rados.rados.0.smithi104.stdout:23: finishing write tid 3 to smithi10428818-23 2020-03-09T00:24:44.872 INFO:tasks.rados.rados.0.smithi104.stdout:25: finishing write tid 1 to smithi10428818-25 2020-03-09T00:24:44.872 INFO:tasks.rados.rados.0.smithi104.stdout:21: finishing write tid 3 to smithi10428818-21 2020-03-09T00:24:44.872 INFO:tasks.rados.rados.0.smithi104.stdout:16: finishing write tid 4 to smithi10428818-16 2020-03-09T00:24:44.872 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-03-09T00:24:44.872 INFO:tasks.rados.rados.0.smithi104.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2020-03-09T00:24:44.872 INFO:tasks.rados.rados.0.smithi104.stdout:23: finishing write tid 4 to smithi10428818-23 2020-03-09T00:24:44.873 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 23 v 4 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-03-09T00:24:44.873 INFO:tasks.rados.rados.0.smithi104.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2020-03-09T00:24:44.873 INFO:tasks.rados.rados.0.smithi104.stdout:26: finishing write tid 3 to smithi10428818-26 2020-03-09T00:24:44.873 INFO:tasks.rados.rados.0.smithi104.stdout:22: finishing write tid 3 to smithi10428818-22 2020-03-09T00:24:44.873 INFO:tasks.rados.rados.0.smithi104.stdout:21: finishing write tid 4 to smithi10428818-21 2020-03-09T00:24:44.873 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 21 v 5 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-03-09T00:24:44.873 INFO:tasks.rados.rados.0.smithi104.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2020-03-09T00:24:44.874 INFO:tasks.rados.rados.0.smithi104.stdout:25: finishing write tid 3 to smithi10428818-25 2020-03-09T00:24:44.874 INFO:tasks.rados.rados.0.smithi104.stdout:19: finishing write tid 4 to smithi10428818-19 2020-03-09T00:24:44.874 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-03-09T00:24:44.874 INFO:tasks.rados.rados.0.smithi104.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2020-03-09T00:24:44.874 INFO:tasks.rados.rados.0.smithi104.stdout:24: finishing write tid 3 to smithi10428818-24 2020-03-09T00:24:44.874 INFO:tasks.rados.rados.0.smithi104.stdout:27: finishing write tid 1 to smithi10428818-27 2020-03-09T00:24:44.875 INFO:tasks.rados.rados.0.smithi104.stdout:14: done (15 left) 2020-03-09T00:24:44.875 INFO:tasks.rados.rados.0.smithi104.stdout:16: done (14 left) 2020-03-09T00:24:44.875 INFO:tasks.rados.rados.0.smithi104.stdout:18: done (13 left) 2020-03-09T00:24:44.875 INFO:tasks.rados.rados.0.smithi104.stdout:19: done (12 left) 2020-03-09T00:24:44.875 INFO:tasks.rados.rados.0.smithi104.stdout:21: done (11 left) 2020-03-09T00:24:44.877 INFO:tasks.rados.rados.0.smithi104.stdout:23: done (10 left) 2020-03-09T00:24:44.877 INFO:tasks.rados.rados.0.smithi104.stdout:33: write initial oid 33 2020-03-09T00:24:44.877 INFO:tasks.rados.rados.0.smithi104.stdout:33: seq_num 32 ranges {0=540672} 2020-03-09T00:24:44.879 INFO:tasks.rados.rados.0.smithi104.stdout:33: writing smithi10428818-33 from 0 to 540672 tid 1 2020-03-09T00:24:44.879 INFO:tasks.rados.rados.0.smithi104.stdout:34: write initial oid 34 2020-03-09T00:24:44.879 INFO:tasks.rados.rados.0.smithi104.stdout:34: seq_num 33 ranges {0=557056} 2020-03-09T00:24:44.880 INFO:tasks.rados.rados.0.smithi104.stdout:34: writing smithi10428818-34 from 0 to 557056 tid 1 2020-03-09T00:24:44.880 INFO:tasks.rados.rados.0.smithi104.stdout:35: write initial oid 35 2020-03-09T00:24:44.880 INFO:tasks.rados.rados.0.smithi104.stdout:35: seq_num 34 ranges {0=573440} 2020-03-09T00:24:44.880 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:44.877+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:44.881 INFO:tasks.rados.rados.0.smithi104.stdout:35: writing smithi10428818-35 from 0 to 573440 tid 1 2020-03-09T00:24:44.881 INFO:tasks.rados.rados.0.smithi104.stdout:36: write initial oid 36 2020-03-09T00:24:44.882 INFO:tasks.rados.rados.0.smithi104.stdout:36: seq_num 35 ranges {0=589824} 2020-03-09T00:24:44.885 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:44.885 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:44.886 INFO:tasks.rados.rados.0.smithi104.stdout:36: writing smithi10428818-36 from 0 to 589824 tid 1 2020-03-09T00:24:44.886 INFO:tasks.rados.rados.0.smithi104.stdout:37: write initial oid 37 2020-03-09T00:24:44.886 INFO:tasks.rados.rados.0.smithi104.stdout:37: seq_num 36 ranges {0=606208} 2020-03-09T00:24:44.889 INFO:tasks.rados.rados.0.smithi104.stdout:37: writing smithi10428818-37 from 0 to 606208 tid 1 2020-03-09T00:24:44.889 INFO:tasks.rados.rados.0.smithi104.stdout:38: write initial oid 38 2020-03-09T00:24:44.889 INFO:tasks.rados.rados.0.smithi104.stdout:38: seq_num 37 ranges {0=622592} 2020-03-09T00:24:44.893 INFO:tasks.rados.rados.0.smithi104.stdout:38: writing smithi10428818-38 from 0 to 622592 tid 1 2020-03-09T00:24:44.894 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.894 INFO:tasks.rados.rados.0.smithi104.stdout:22: finishing write tid 4 to smithi10428818-22 2020-03-09T00:24:44.895 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 22 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-03-09T00:24:44.895 INFO:tasks.rados.rados.0.smithi104.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2020-03-09T00:24:44.895 INFO:tasks.rados.rados.0.smithi104.stdout:25: finishing write tid 4 to smithi10428818-25 2020-03-09T00:24:44.896 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-03-09T00:24:44.896 INFO:tasks.rados.rados.0.smithi104.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2020-03-09T00:24:44.896 INFO:tasks.rados.rados.0.smithi104.stdout:30: finishing write tid 1 to smithi10428818-30 2020-03-09T00:24:44.896 INFO:tasks.rados.rados.0.smithi104.stdout:22: done (15 left) 2020-03-09T00:24:44.897 INFO:tasks.rados.rados.0.smithi104.stdout:25: done (14 left) 2020-03-09T00:24:44.897 INFO:tasks.rados.rados.0.smithi104.stdout:39: write initial oid 39 2020-03-09T00:24:44.897 INFO:tasks.rados.rados.0.smithi104.stdout:39: seq_num 38 ranges {0=638976} 2020-03-09T00:24:44.899 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:24:44.899 INFO:tasks.rados.rados.0.smithi104.stdout:39: writing smithi10428818-39 from 0 to 638976 tid 1 2020-03-09T00:24:44.899 INFO:tasks.rados.rados.0.smithi104.stdout:40: write initial oid 40 2020-03-09T00:24:44.899 INFO:tasks.rados.rados.0.smithi104.stdout:40: seq_num 39 ranges {0=671744} 2020-03-09T00:24:44.903 INFO:tasks.rados.rados.0.smithi104.stdout:40: writing smithi10428818-40 from 0 to 671744 tid 1 2020-03-09T00:24:44.903 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.903 INFO:tasks.rados.rados.0.smithi104.stdout:27: finishing write tid 3 to smithi10428818-27 2020-03-09T00:24:44.903 INFO:tasks.rados.rados.0.smithi104.stdout:24: finishing write tid 4 to smithi10428818-24 2020-03-09T00:24:44.903 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-03-09T00:24:44.904 INFO:tasks.rados.rados.0.smithi104.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2020-03-09T00:24:44.904 INFO:tasks.rados.rados.0.smithi104.stdout:26: finishing write tid 4 to smithi10428818-26 2020-03-09T00:24:44.904 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-03-09T00:24:44.904 INFO:tasks.rados.rados.0.smithi104.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2020-03-09T00:24:44.904 INFO:tasks.rados.rados.0.smithi104.stdout:29: finishing write tid 1 to smithi10428818-29 2020-03-09T00:24:44.904 INFO:tasks.rados.rados.0.smithi104.stdout:27: finishing write tid 4 to smithi10428818-27 2020-03-09T00:24:44.904 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 27 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-03-09T00:24:44.905 INFO:tasks.rados.rados.0.smithi104.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2020-03-09T00:24:44.905 INFO:tasks.rados.rados.0.smithi104.stdout:29: finishing write tid 3 to smithi10428818-29 2020-03-09T00:24:44.905 INFO:tasks.rados.rados.0.smithi104.stdout:30: finishing write tid 3 to smithi10428818-30 2020-03-09T00:24:44.905 INFO:tasks.rados.rados.0.smithi104.stdout:30: finishing write tid 4 to smithi10428818-30 2020-03-09T00:24:44.905 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-03-09T00:24:44.905 INFO:tasks.rados.rados.0.smithi104.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2020-03-09T00:24:44.905 INFO:tasks.rados.rados.0.smithi104.stdout:29: finishing write tid 4 to smithi10428818-29 2020-03-09T00:24:44.906 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 29 v 8 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-03-09T00:24:44.906 INFO:tasks.rados.rados.0.smithi104.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2020-03-09T00:24:44.906 INFO:tasks.rados.rados.0.smithi104.stdout:31: finishing write tid 1 to smithi10428818-31 2020-03-09T00:24:44.906 INFO:tasks.rados.rados.0.smithi104.stdout:32: finishing write tid 1 to smithi10428818-32 2020-03-09T00:24:44.906 INFO:tasks.rados.rados.0.smithi104.stdout:31: finishing write tid 3 to smithi10428818-31 2020-03-09T00:24:44.906 INFO:tasks.rados.rados.0.smithi104.stdout:35: finishing write tid 1 to smithi10428818-35 2020-03-09T00:24:44.906 INFO:tasks.rados.rados.0.smithi104.stdout:32: finishing write tid 3 to smithi10428818-32 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:33: finishing write tid 1 to smithi10428818-33 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:36: finishing write tid 1 to smithi10428818-36 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:32: finishing write tid 4 to smithi10428818-32 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:35: finishing write tid 3 to smithi10428818-35 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:31: finishing write tid 4 to smithi10428818-31 2020-03-09T00:24:44.907 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 31 v 2 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-03-09T00:24:44.908 INFO:tasks.rados.rados.0.smithi104.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2020-03-09T00:24:44.908 INFO:tasks.rados.rados.0.smithi104.stdout:34: finishing write tid 1 to smithi10428818-34 2020-03-09T00:24:44.908 INFO:tasks.rados.rados.0.smithi104.stdout:37: finishing write tid 1 to smithi10428818-37 2020-03-09T00:24:44.908 INFO:tasks.rados.rados.0.smithi104.stdout:35: finishing write tid 4 to smithi10428818-35 2020-03-09T00:24:44.908 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 35 v 10 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-03-09T00:24:44.908 INFO:tasks.rados.rados.0.smithi104.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2020-03-09T00:24:44.908 INFO:tasks.rados.rados.0.smithi104.stdout:24: done (15 left) 2020-03-09T00:24:44.909 INFO:tasks.rados.rados.0.smithi104.stdout:26: done (14 left) 2020-03-09T00:24:44.909 INFO:tasks.rados.rados.0.smithi104.stdout:27: done (13 left) 2020-03-09T00:24:44.909 INFO:tasks.rados.rados.0.smithi104.stdout:29: done (12 left) 2020-03-09T00:24:44.909 INFO:tasks.rados.rados.0.smithi104.stdout:30: done (11 left) 2020-03-09T00:24:44.909 INFO:tasks.rados.rados.0.smithi104.stdout:31: done (10 left) 2020-03-09T00:24:44.909 INFO:tasks.rados.rados.0.smithi104.stdout:32: done (9 left) 2020-03-09T00:24:44.909 INFO:tasks.rados.rados.0.smithi104.stdout:35: done (8 left) 2020-03-09T00:24:44.910 INFO:tasks.rados.rados.0.smithi104.stdout:41: write initial oid 41 2020-03-09T00:24:44.910 INFO:tasks.rados.rados.0.smithi104.stdout:41: seq_num 40 ranges {0=688128} 2020-03-09T00:24:44.910 INFO:tasks.rados.rados.0.smithi104.stdout:41: writing smithi10428818-41 from 0 to 688128 tid 1 2020-03-09T00:24:44.910 INFO:tasks.rados.rados.0.smithi104.stdout:42: write initial oid 42 2020-03-09T00:24:44.911 INFO:tasks.rados.rados.0.smithi104.stdout:42: seq_num 41 ranges {0=704512} 2020-03-09T00:24:44.912 INFO:tasks.rados.rados.0.smithi104.stdout:42: writing smithi10428818-42 from 0 to 704512 tid 1 2020-03-09T00:24:44.912 INFO:tasks.rados.rados.0.smithi104.stdout:43: write initial oid 43 2020-03-09T00:24:44.912 INFO:tasks.rados.rados.0.smithi104.stdout:43: seq_num 42 ranges {0=720896} 2020-03-09T00:24:44.917 INFO:tasks.rados.rados.0.smithi104.stdout:43: writing smithi10428818-43 from 0 to 720896 tid 1 2020-03-09T00:24:44.917 INFO:tasks.rados.rados.0.smithi104.stdout:44: write initial oid 44 2020-03-09T00:24:44.918 INFO:tasks.rados.rados.0.smithi104.stdout:44: seq_num 43 ranges {0=737280} 2020-03-09T00:24:44.922 INFO:tasks.rados.rados.0.smithi104.stdout:44: writing smithi10428818-44 from 0 to 737280 tid 1 2020-03-09T00:24:44.922 INFO:tasks.rados.rados.0.smithi104.stdout:45: write initial oid 45 2020-03-09T00:24:44.922 INFO:tasks.rados.rados.0.smithi104.stdout:45: seq_num 44 ranges {0=753664} 2020-03-09T00:24:44.927 INFO:tasks.rados.rados.0.smithi104.stdout:45: writing smithi10428818-45 from 0 to 753664 tid 1 2020-03-09T00:24:44.928 INFO:tasks.rados.rados.0.smithi104.stdout:46: write initial oid 46 2020-03-09T00:24:44.928 INFO:tasks.rados.rados.0.smithi104.stdout:46: seq_num 45 ranges {0=770048} 2020-03-09T00:24:44.932 INFO:tasks.rados.rados.0.smithi104.stdout:46: writing smithi10428818-46 from 0 to 770048 tid 1 2020-03-09T00:24:44.932 INFO:tasks.rados.rados.0.smithi104.stdout:47: write initial oid 47 2020-03-09T00:24:44.932 INFO:tasks.rados.rados.0.smithi104.stdout:47: seq_num 46 ranges {0=786432} 2020-03-09T00:24:44.937 INFO:tasks.rados.rados.0.smithi104.stdout:47: writing smithi10428818-47 from 0 to 786432 tid 1 2020-03-09T00:24:44.937 INFO:tasks.rados.rados.0.smithi104.stdout:48: write initial oid 48 2020-03-09T00:24:44.937 INFO:tasks.rados.rados.0.smithi104.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2020-03-09T00:24:44.939 INFO:tasks.rados.rados.0.smithi104.stdout:48: writing smithi10428818-48 from 0 to 425984 tid 1 2020-03-09T00:24:44.943 INFO:tasks.rados.rados.0.smithi104.stdout:48: writing smithi10428818-48 from 425984 to 802816 tid 2 2020-03-09T00:24:44.943 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 16 2020-03-09T00:24:44.943 INFO:tasks.rados.rados.0.smithi104.stdout:38: finishing write tid 1 to smithi10428818-38 2020-03-09T00:24:44.943 INFO:tasks.rados.rados.0.smithi104.stdout:33: finishing write tid 3 to smithi10428818-33 2020-03-09T00:24:44.943 INFO:tasks.rados.rados.0.smithi104.stdout:34: finishing write tid 3 to smithi10428818-34 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:36: finishing write tid 3 to smithi10428818-36 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:33: finishing write tid 4 to smithi10428818-33 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:34: finishing write tid 4 to smithi10428818-34 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 34 v 13 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2020-03-09T00:24:44.944 INFO:tasks.rados.rados.0.smithi104.stdout:37: finishing write tid 3 to smithi10428818-37 2020-03-09T00:24:44.945 INFO:tasks.rados.rados.0.smithi104.stdout:38: finishing write tid 3 to smithi10428818-38 2020-03-09T00:24:44.945 INFO:tasks.rados.rados.0.smithi104.stdout:39: finishing write tid 1 to smithi10428818-39 2020-03-09T00:24:44.945 INFO:tasks.rados.rados.0.smithi104.stdout:36: finishing write tid 4 to smithi10428818-36 2020-03-09T00:24:44.945 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 36 v 2 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-03-09T00:24:44.945 INFO:tasks.rados.rados.0.smithi104.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2020-03-09T00:24:44.945 INFO:tasks.rados.rados.0.smithi104.stdout:37: finishing write tid 4 to smithi10428818-37 2020-03-09T00:24:44.945 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 37 v 14 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-03-09T00:24:44.946 INFO:tasks.rados.rados.0.smithi104.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:24:44.946 INFO:tasks.rados.rados.0.smithi104.stdout:39: finishing write tid 3 to smithi10428818-39 2020-03-09T00:24:44.946 INFO:tasks.rados.rados.0.smithi104.stdout:40: finishing write tid 1 to smithi10428818-40 2020-03-09T00:24:44.946 INFO:tasks.rados.rados.0.smithi104.stdout:38: finishing write tid 4 to smithi10428818-38 2020-03-09T00:24:44.946 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 38 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-03-09T00:24:44.946 INFO:tasks.rados.rados.0.smithi104.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:41: finishing write tid 1 to smithi10428818-41 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:40: finishing write tid 3 to smithi10428818-40 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:39: finishing write tid 4 to smithi10428818-39 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:42: finishing write tid 1 to smithi10428818-42 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:40: finishing write tid 4 to smithi10428818-40 2020-03-09T00:24:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 40 v 4 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-03-09T00:24:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2020-03-09T00:24:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:41: finishing write tid 3 to smithi10428818-41 2020-03-09T00:24:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:42: finishing write tid 3 to smithi10428818-42 2020-03-09T00:24:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:41: finishing write tid 4 to smithi10428818-41 2020-03-09T00:24:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 41 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-03-09T00:24:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2020-03-09T00:24:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:43: finishing write tid 1 to smithi10428818-43 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:42: finishing write tid 4 to smithi10428818-42 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 42 v 8 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:44: finishing write tid 1 to smithi10428818-44 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:45: finishing write tid 1 to smithi10428818-45 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:46: finishing write tid 1 to smithi10428818-46 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:33: done (15 left) 2020-03-09T00:24:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:34: done (14 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:36: done (13 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:37: done (12 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:38: done (11 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:39: done (10 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:40: done (9 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:41: done (8 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:42: done (7 left) 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:49: write initial oid 49 2020-03-09T00:24:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2020-03-09T00:24:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:49: writing smithi10428818-49 from 0 to 442368 tid 1 2020-03-09T00:24:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:49: writing smithi10428818-49 from 442368 to 819200 tid 2 2020-03-09T00:24:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:50: write initial oid 50 2020-03-09T00:24:44.952 INFO:tasks.rados.rados.0.smithi104.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2020-03-09T00:24:44.952 INFO:tasks.rados.rados.0.smithi104.stdout:50: writing smithi10428818-50 from 0 to 458752 tid 1 2020-03-09T00:24:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:50: writing smithi10428818-50 from 458752 to 835584 tid 2 2020-03-09T00:24:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:51: delete oid 40 current snap is 0 2020-03-09T00:24:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:44: finishing write tid 3 to smithi10428818-44 2020-03-09T00:24:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:43: finishing write tid 3 to smithi10428818-43 2020-03-09T00:24:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:45: finishing write tid 3 to smithi10428818-45 2020-03-09T00:24:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:46: finishing write tid 3 to smithi10428818-46 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:47: finishing write tid 1 to smithi10428818-47 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:48: finishing write tid 1 to smithi10428818-48 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:44: finishing write tid 4 to smithi10428818-44 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:46: finishing write tid 4 to smithi10428818-46 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-03-09T00:24:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2020-03-09T00:24:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:43: finishing write tid 4 to smithi10428818-43 2020-03-09T00:24:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 43 v 5 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-03-09T00:24:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2020-03-09T00:24:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:45: finishing write tid 4 to smithi10428818-45 2020-03-09T00:24:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-03-09T00:24:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2020-03-09T00:24:44.956 INFO:tasks.rados.rados.0.smithi104.stdout:49: finishing write tid 1 to smithi10428818-49 2020-03-09T00:24:44.956 INFO:tasks.rados.rados.0.smithi104.stdout:47: finishing write tid 3 to smithi10428818-47 2020-03-09T00:24:44.956 INFO:tasks.rados.rados.0.smithi104.stdout:48: finishing write tid 2 to smithi10428818-48 2020-03-09T00:24:44.956 INFO:tasks.rados.rados.0.smithi104.stdout:48: finishing write tid 4 to smithi10428818-48 2020-03-09T00:24:44.959 INFO:tasks.rados.rados.0.smithi104.stdout:50: finishing write tid 1 to smithi10428818-50 2020-03-09T00:24:44.959 INFO:tasks.rados.rados.0.smithi104.stdout:48: finishing write tid 5 to smithi10428818-48 2020-03-09T00:24:44.960 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-03-09T00:24:44.960 INFO:tasks.rados.rados.0.smithi104.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2020-03-09T00:24:44.960 INFO:tasks.rados.rados.0.smithi104.stdout:47: finishing write tid 4 to smithi10428818-47 2020-03-09T00:24:44.960 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 47 v 10 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-03-09T00:24:44.961 INFO:tasks.rados.rados.0.smithi104.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2020-03-09T00:24:44.961 INFO:tasks.rados.rados.0.smithi104.stdout:43: done (9 left) 2020-03-09T00:24:44.962 INFO:tasks.rados.rados.0.smithi104.stdout:44: done (8 left) 2020-03-09T00:24:44.962 INFO:tasks.rados.rados.0.smithi104.stdout:45: done (7 left) 2020-03-09T00:24:44.962 INFO:tasks.rados.rados.0.smithi104.stdout:46: done (6 left) 2020-03-09T00:24:44.962 INFO:tasks.rados.rados.0.smithi104.stdout:47: done (5 left) 2020-03-09T00:24:44.962 INFO:tasks.rados.rados.0.smithi104.stdout:48: done (4 left) 2020-03-09T00:24:44.962 INFO:tasks.rados.rados.0.smithi104.stdout:51: done (3 left) 2020-03-09T00:24:44.962 INFO:tasks.rados.rados.0.smithi104.stdout:52: read oid 13 snap -1 2020-03-09T00:24:44.963 INFO:tasks.rados.rados.0.smithi104.stdout:52: expect (ObjNum 12 snap 0 seq_num 12) 2020-03-09T00:24:44.963 INFO:tasks.rados.rados.0.smithi104.stdout:53: delete oid 21 current snap is 0 2020-03-09T00:24:44.963 INFO:tasks.rados.rados.0.smithi104.stdout:49: finishing write tid 2 to smithi10428818-49 2020-03-09T00:24:44.963 INFO:tasks.rados.rados.0.smithi104.stdout:49: finishing write tid 4 to smithi10428818-49 2020-03-09T00:24:44.967 INFO:tasks.rados.rados.0.smithi104.stdout:53: done (4 left) 2020-03-09T00:24:44.967 INFO:tasks.rados.rados.0.smithi104.stdout:54: read oid 12 snap -1 2020-03-09T00:24:44.967 INFO:tasks.rados.rados.0.smithi104.stdout:54: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:24:44.967 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 10 current snap is 0 2020-03-09T00:24:44.968 INFO:tasks.rados.rados.0.smithi104.stdout:55: seq_num 52 ranges {163840=507904,671744=376832} 2020-03-09T00:24:44.971 INFO:tasks.rados.rados.0.smithi104.stdout:55: writing smithi10428818-10 from 163840 to 671744 tid 1 2020-03-09T00:24:44.973 INFO:tasks.rados.rados.0.smithi104.stdout:55: writing smithi10428818-10 from 671744 to 1048576 tid 2 2020-03-09T00:24:44.973 INFO:tasks.rados.rados.0.smithi104.stdout:56: read oid 45 snap -1 2020-03-09T00:24:44.974 INFO:tasks.rados.rados.0.smithi104.stdout:56: expect (ObjNum 44 snap 0 seq_num 44) 2020-03-09T00:24:44.974 INFO:tasks.rados.rados.0.smithi104.stdout:57: rollback oid 35 current snap is 0 2020-03-09T00:24:44.974 INFO:tasks.rados.rados.0.smithi104.stdout:57: done (7 left) 2020-03-09T00:24:44.974 INFO:tasks.rados.rados.0.smithi104.stdout:58: snap_create 2020-03-09T00:24:44.974 INFO:tasks.rados.rados.0.smithi104.stdout:50: finishing write tid 2 to smithi10428818-50 2020-03-09T00:24:44.974 INFO:tasks.rados.rados.0.smithi104.stdout:52: expect (ObjNum 12 snap 0 seq_num 12) 2020-03-09T00:24:44.979 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:44.977+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:44.980 INFO:tasks.rados.rados.0.smithi104.stdout:49: finishing write tid 5 to smithi10428818-49 2020-03-09T00:24:44.980 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-03-09T00:24:44.980 INFO:tasks.rados.rados.0.smithi104.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2020-03-09T00:24:44.980 INFO:tasks.rados.rados.0.smithi104.stdout:50: finishing write tid 4 to smithi10428818-50 2020-03-09T00:24:44.981 INFO:tasks.rados.rados.0.smithi104.stdout:50: finishing write tid 5 to smithi10428818-50 2020-03-09T00:24:44.981 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-03-09T00:24:44.981 INFO:tasks.rados.rados.0.smithi104.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2020-03-09T00:24:44.981 INFO:tasks.rados.rados.0.smithi104.stdout:54: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:24:44.985 INFO:tasks.rados.rados.0.smithi104.stdout:55: finishing write tid 1 to smithi10428818-10 2020-03-09T00:24:44.985 INFO:tasks.rados.rados.0.smithi104.stdout:55: finishing write tid 2 to smithi10428818-10 2020-03-09T00:24:44.985 INFO:tasks.rados.rados.0.smithi104.stdout:56: expect (ObjNum 44 snap 0 seq_num 44) 2020-03-09T00:24:45.004 INFO:tasks.rados.rados.0.smithi104.stdout:55: finishing write tid 4 to smithi10428818-10 2020-03-09T00:24:45.004 INFO:tasks.rados.rados.0.smithi104.stdout:55: finishing write tid 5 to smithi10428818-10 2020-03-09T00:24:45.004 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 10 v 10 (ObjNum 52 snap 0 seq_num 52) dirty exists 2020-03-09T00:24:45.004 INFO:tasks.rados.rados.0.smithi104.stdout:55: left oid 10 (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:24:45.062 INFO:tasks.rados.rados.0.smithi104.stdout:28: finishing write tid 1 to smithi10428818-28 2020-03-09T00:24:45.065 INFO:tasks.rados.rados.0.smithi104.stdout:28: finishing write tid 3 to smithi10428818-28 2020-03-09T00:24:45.067 INFO:tasks.rados.rados.0.smithi104.stdout:28: finishing write tid 4 to smithi10428818-28 2020-03-09T00:24:45.067 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 28 v 2 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-03-09T00:24:45.067 INFO:tasks.rados.rados.0.smithi104.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2020-03-09T00:24:45.080 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:45.080 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:45.080 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:45.079+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:45.089 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:24:45.181 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:45.179+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:45.205 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:45.206 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:45.214 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:24:45.283 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:45.281+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:45.330 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:45.330 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:45.338 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:24:45.384 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:45.382+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:45.453 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:45.453 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:45.462 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:24:45.485 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:45.483+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:45.580 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:45.580 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:45.586 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:45.584+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:45.588 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:24:45.687 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:45.684+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:45.704 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:45.704 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:45.712 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:24:45.788 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:45.786+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:45.802 INFO:tasks.rados.rados.0.smithi104.stdout:28: done (7 left) 2020-03-09T00:24:45.802 INFO:tasks.rados.rados.0.smithi104.stdout:49: done (6 left) 2020-03-09T00:24:45.802 INFO:tasks.rados.rados.0.smithi104.stdout:50: done (5 left) 2020-03-09T00:24:45.802 INFO:tasks.rados.rados.0.smithi104.stdout:52: done (4 left) 2020-03-09T00:24:45.802 INFO:tasks.rados.rados.0.smithi104.stdout:54: done (3 left) 2020-03-09T00:24:45.802 INFO:tasks.rados.rados.0.smithi104.stdout:55: done (2 left) 2020-03-09T00:24:45.802 INFO:tasks.rados.rados.0.smithi104.stdout:56: done (1 left) 2020-03-09T00:24:45.803 INFO:tasks.rados.rados.0.smithi104.stdout:58: done (0 left) 2020-03-09T00:24:45.803 INFO:tasks.rados.rados.0.smithi104.stdout:59: snap_remove snap 0 2020-03-09T00:24:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is unset 2020-03-09T00:24:45.868 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:45.868 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:45.877 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:24:45.921 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:45.920+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:45.990 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:45.988+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:46.004 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:46.004 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:46.015 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:24:46.099 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:46.098+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:46.173 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:46.173 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:46.182 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:24:46.223 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:46.222+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:46.292 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:46.290+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:46.333 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:24:46.338 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:46.338 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:46.346 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:24:46.393 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:46.392+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:46.468 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:46.483 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:46.484 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:46.488 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:46.493 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:24:46.496 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:46.495+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:46.508 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:46.528 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:46.548 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:46.568 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:46.588 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:46.597 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:46.595+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:46.608 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:46.619 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:46.620 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:46.628 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:24:46.629 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:46.648 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:46.668 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:46.688 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:46.698 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:46.697+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:46.708 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:46.728 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:46.748 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:46.754 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:46.754 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:24:46.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:24:46.754 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:24:46.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.754 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:1 1.0 1:e8e3cb91:::INTEL_SSDPED1D480GA_PHMB7513001F480DGN:head [omap-get-keys] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-03-09T00:24:46.754 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.661953+0000", 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.090194705999998, 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030079800000000002, 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:24:46.755 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.661953+0000", 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.661953+0000", 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.723e-05 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.662001+0000", 2020-03-09T00:24:46.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4158999999999999e-05 2020-03-09T00:24:46.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.662015+0000", 2020-03-09T00:24:46.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3787e-05 2020-03-09T00:24:46.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.662028+0000", 2020-03-09T00:24:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0480000000000004e-06 2020-03-09T00:24:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.662038+0000", 2020-03-09T00:24:46.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5716999999999994e-05 2020-03-09T00:24:46.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.662103+0000", 2020-03-09T00:24:46.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015085700000000001 2020-03-09T00:24:46.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.662254+0000", 2020-03-09T00:24:46.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030079800000000002 2020-03-09T00:24:46.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.761 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:2 1.0 1:e8e3cb91:::INTEL_SSDPED1D480GA_PHMB7513001F480DGN:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.663306+0000", 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.088841676000001, 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018269950000000001, 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.762 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663306+0000", 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663306+0000", 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8458000000000002e-05 2020-03-09T00:24:46.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663365+0000", 2020-03-09T00:24:46.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9190000000000004e-06 2020-03-09T00:24:46.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663371+0000", 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2291e-05 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663383+0000", 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6699999999999994e-06 2020-03-09T00:24:46.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663391+0000", 2020-03-09T00:24:46.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0685e-05 2020-03-09T00:24:46.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663441+0000", 2020-03-09T00:24:46.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000138608 2020-03-09T00:24:46.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:46.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663580+0000", 2020-03-09T00:24:46.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013393300000000001 2020-03-09T00:24:46.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0", 2020-03-09T00:24:46.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663714+0000", 2020-03-09T00:24:46.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052014900000000002 2020-03-09T00:24:46.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2020-03-09T00:24:46.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664234+0000", 2020-03-09T00:24:46.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080904499999999999 2020-03-09T00:24:46.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665043+0000", 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1595999999999999e-05 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665095+0000", 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8640999999999998e-05 2020-03-09T00:24:46.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665133+0000", 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018269950000000001 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4118.0:2 1.0 e26/24 ondisk, result = 0)", 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.664906+0000", 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.087242200999999, 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023222999999999999, 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:24:46.773 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664906+0000", 2020-03-09T00:24:46.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664906+0000", 2020-03-09T00:24:46.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3155999999999999e-05 2020-03-09T00:24:46.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664939+0000", 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5339999999999996e-06 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664946+0000", 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.075e-05 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664956+0000", 2020-03-09T00:24:46.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9210000000000003e-06 2020-03-09T00:24:46.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664963+0000", 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2335999999999994e-05 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665026+0000", 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4339999999999998e-06 2020-03-09T00:24:46.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:46.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665035+0000", 2020-03-09T00:24:46.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103099 2020-03-09T00:24:46.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665138+0000", 2020-03-09T00:24:46.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023222999999999999 2020-03-09T00:24:46.780 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.780 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.780 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.781 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:3 1.0 1:a3830038:::INTEL_SSDPEDMD400G4_PHFT620500N6400BGN:head [omap-get-keys] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-03-09T00:24:46.781 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.745006+0000", 2020-03-09T00:24:46.781 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.007142511000001, 2020-03-09T00:24:46.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000255412, 2020-03-09T00:24:46.781 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.781 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:46.781 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:46.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:24:46.782 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:24:46.782 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3 2020-03-09T00:24:46.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.782 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745006+0000", 2020-03-09T00:24:46.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745006+0000", 2020-03-09T00:24:46.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7256000000000003e-05 2020-03-09T00:24:46.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745053+0000", 2020-03-09T00:24:46.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4469999999999997e-06 2020-03-09T00:24:46.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745058+0000", 2020-03-09T00:24:46.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2660000000000001e-05 2020-03-09T00:24:46.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745071+0000", 2020-03-09T00:24:46.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2119999999999997e-06 2020-03-09T00:24:46.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745078+0000", 2020-03-09T00:24:46.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0520999999999998e-05 2020-03-09T00:24:46.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745149+0000", 2020-03-09T00:24:46.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112316 2020-03-09T00:24:46.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745261+0000", 2020-03-09T00:24:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000255412 2020-03-09T00:24:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:4 1.0 1:a3830038:::INTEL_SSDPEDMD400G4_PHFT620500N6400BGN:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.745925+0000", 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.006223405, 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018936129999999999, 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.790 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:46.791 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:46.791 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:24:46.791 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:24:46.791 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 4 2020-03-09T00:24:46.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.791 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745925+0000", 2020-03-09T00:24:46.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745925+0000", 2020-03-09T00:24:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1280999999999998e-05 2020-03-09T00:24:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745986+0000", 2020-03-09T00:24:46.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3940000000000004e-06 2020-03-09T00:24:46.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745991+0000", 2020-03-09T00:24:46.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1146e-05 2020-03-09T00:24:46.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746002+0000", 2020-03-09T00:24:46.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9070000000000007e-06 2020-03-09T00:24:46.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746010+0000", 2020-03-09T00:24:46.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2175000000000001e-05 2020-03-09T00:24:46.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746063+0000", 2020-03-09T00:24:46.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013140800000000001 2020-03-09T00:24:46.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746194+0000", 2020-03-09T00:24:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010697 2020-03-09T00:24:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0", 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746301+0000", 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000434876 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746736+0000", 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099432899999999996 2020-03-09T00:24:46.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747730+0000", 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9877999999999999e-05 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:46.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747780+0000", 2020-03-09T00:24:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8248999999999999e-05 2020-03-09T00:24:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747818+0000", 2020-03-09T00:24:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018936129999999999 2020-03-09T00:24:46.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.801 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.801 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.801 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4118.0:4 1.0 e26/24 ondisk, result = 0)", 2020-03-09T00:24:46.801 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.747593+0000", 2020-03-09T00:24:46.801 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.004554687999999, 2020-03-09T00:24:46.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000229137, 2020-03-09T00:24:46.802 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.802 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:24:46.802 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747593+0000", 2020-03-09T00:24:46.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747593+0000", 2020-03-09T00:24:46.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7002000000000002e-05 2020-03-09T00:24:46.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747630+0000", 2020-03-09T00:24:46.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6629999999999998e-06 2020-03-09T00:24:46.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747636+0000", 2020-03-09T00:24:46.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0924000000000001e-05 2020-03-09T00:24:46.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747647+0000", 2020-03-09T00:24:46.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5016e-05 2020-03-09T00:24:46.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747662+0000", 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5971999999999999e-05 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747718+0000", 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6209999999999998e-06 2020-03-09T00:24:46.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747725+0000", 2020-03-09T00:24:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7938999999999999e-05 2020-03-09T00:24:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747822+0000", 2020-03-09T00:24:46.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000229137 2020-03-09T00:24:46.809 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.809 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.811 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:5 1.0 1:15e1a584:::INTEL_SSDPEDMD400G4_CVFT6233005G400BGN:head [omap-get-keys] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-03-09T00:24:46.811 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.842811+0000", 2020-03-09T00:24:46.811 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.909337368999999, 2020-03-09T00:24:46.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025693599999999998, 2020-03-09T00:24:46.811 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.811 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.842811+0000", 2020-03-09T00:24:46.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.842811+0000", 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7978e-05 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.842859+0000", 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3700000000000003e-06 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.842864+0000", 2020-03-09T00:24:46.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2694000000000001e-05 2020-03-09T00:24:46.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.842877+0000", 2020-03-09T00:24:46.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2239999999999998e-06 2020-03-09T00:24:46.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.842884+0000", 2020-03-09T00:24:46.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1202e-05 2020-03-09T00:24:46.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.842955+0000", 2020-03-09T00:24:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011246799999999999 2020-03-09T00:24:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843068+0000", 2020-03-09T00:24:46.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025693599999999998 2020-03-09T00:24:46.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.818 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.819 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:6 1.0 1:15e1a584:::INTEL_SSDPEDMD400G4_CVFT6233005G400BGN:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-03-09T00:24:46.819 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.843698+0000", 2020-03-09T00:24:46.819 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.908450251000001, 2020-03-09T00:24:46.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020264290000000002, 2020-03-09T00:24:46.819 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.819 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:46.820 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:46.820 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:24:46.820 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:24:46.820 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2020-03-09T00:24:46.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.820 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843698+0000", 2020-03-09T00:24:46.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843698+0000", 2020-03-09T00:24:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4163000000000003e-05 2020-03-09T00:24:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843752+0000", 2020-03-09T00:24:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9470000000000002e-06 2020-03-09T00:24:46.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843757+0000", 2020-03-09T00:24:46.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1093999999999999e-05 2020-03-09T00:24:46.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843768+0000", 2020-03-09T00:24:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4360000000000004e-06 2020-03-09T00:24:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843772+0000", 2020-03-09T00:24:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1758999999999997e-05 2020-03-09T00:24:46.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843824+0000", 2020-03-09T00:24:46.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124189 2020-03-09T00:24:46.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:46.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843948+0000", 2020-03-09T00:24:46.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109038 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0", 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.844057+0000", 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073162100000000001 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2020-03-09T00:24:46.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.844789+0000", 2020-03-09T00:24:46.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000853398 2020-03-09T00:24:46.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2020-03-09T00:24:46.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845642+0000", 2020-03-09T00:24:46.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0244000000000002e-05 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845693+0000", 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.154e-05 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845724+0000", 2020-03-09T00:24:46.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020264290000000002 2020-03-09T00:24:46.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.4118.0:6 1.0 e26/24 ondisk, result = 0)", 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.845497+0000", 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.906650769999999, 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023296600000000001, 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:46.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845497+0000", 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845497+0000", 2020-03-09T00:24:46.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8588999999999999e-05 2020-03-09T00:24:46.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:46.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845536+0000", 2020-03-09T00:24:46.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4070000000000001e-06 2020-03-09T00:24:46.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845541+0000", 2020-03-09T00:24:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6819999999999992e-06 2020-03-09T00:24:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:46.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845550+0000", 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7710000000000001e-06 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845557+0000", 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7321000000000001e-05 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:46.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845624+0000", 2020-03-09T00:24:46.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0680000000000003e-06 2020-03-09T00:24:46.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:46.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845633+0000", 2020-03-09T00:24:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7127999999999996e-05 2020-03-09T00:24:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845730+0000", 2020-03-09T00:24:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023296600000000001 2020-03-09T00:24:46.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.838 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:46.838 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:46.838 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:46.841 INFO:tasks.rados.rados.0.smithi104.stdout:59: done (0 left) 2020-03-09T00:24:46.841 INFO:tasks.rados.rados.0.smithi104.stdout:60: read oid 30 snap -1 2020-03-09T00:24:46.841 INFO:tasks.rados.rados.0.smithi104.stdout:60: expect (ObjNum 29 snap 0 seq_num 29) 2020-03-09T00:24:46.841 INFO:tasks.rados.rados.0.smithi104.stdout:61: read oid 12 snap -1 2020-03-09T00:24:46.842 INFO:tasks.rados.rados.0.smithi104.stdout:61: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:24:46.842 INFO:tasks.rados.rados.0.smithi104.stdout:62: read oid 9 snap -1 2020-03-09T00:24:46.842 INFO:tasks.rados.rados.0.smithi104.stdout:62: expect (ObjNum 8 snap 0 seq_num 8) 2020-03-09T00:24:46.842 INFO:tasks.rados.rados.0.smithi104.stdout:63: snap_create 2020-03-09T00:24:46.842 INFO:tasks.rados.rados.0.smithi104.stdout:62: expect (ObjNum 8 snap 0 seq_num 8) 2020-03-09T00:24:46.842 INFO:tasks.rados.rados.0.smithi104.stdout:61: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:24:46.842 INFO:tasks.rados.rados.0.smithi104.stdout:60: expect (ObjNum 29 snap 0 seq_num 29) 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:46.767+0000 7f2c6ed9d700 0 -- 172.21.15.148:0/2749389710 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7f2c70188d20 msgr2=0x7f2c7018b130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.843 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.844 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.844 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.844 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.844 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.844 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.844 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.844 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.845 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.845 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:24:46.845 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:46.845 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:46.838+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:46.846 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:24:46.940 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:46.939+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:46.997 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:46.997 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:46.997 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:24:46.997 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:47.005 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:24:47.041 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:47.039+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:47.142 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:47.140+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:47.161 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:24:47.161 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:24:47.161 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:47.170 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:24:47.253 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:47.252+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:47.325 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:47.325 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:24:47.326 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:24:47.326 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [] 2020-03-09T00:24:47.326 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:47.334 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:24:47.344 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:47.342+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:47.445 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:47.443+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:47.461 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:47.461 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:47.461 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:24:47.462 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:47.470 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:24:47.546 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:47.544+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:47.584 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:47.585 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:47.585 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:24:47.585 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:24:47.585 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:47.593 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:24:47.646 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:47.644+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:47.708 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:47.708 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:24:47.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:24:47.709 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:24:47.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.709 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s3 37/32 ECSubRead(tid=33, to_read={3:c607ec57:::smithi10428818-30:head=0,122880,0}, subchunks={3:c607ec57:::smithi10428818-30:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:47.709 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:46.811273+0000", 2020-03-09T00:24:47.709 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.89620337000000005, 2020-03-09T00:24:47.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016459059999999999, 2020-03-09T00:24:47.709 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:47.710 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:47.710 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:47.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:47.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811273+0000", 2020-03-09T00:24:47.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:47.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811273+0000", 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4289000000000001e-05 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:47.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811327+0000", 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1035e-05 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811338+0000", 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0769000000000001e-05 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:47.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811359+0000", 2020-03-09T00:24:47.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.9369999999999999e-06 2020-03-09T00:24:47.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:47.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811369+0000", 2020-03-09T00:24:47.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00051576499999999995 2020-03-09T00:24:47.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:47.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811885+0000", 2020-03-09T00:24:47.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010341110000000001 2020-03-09T00:24:47.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812919+0000", 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016459059999999999 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:47.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.716 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s3 37/32 ECSubRead(tid=34, to_read={3:c607ec57:::smithi10428818-30:head=0,122880,0}, subchunks={3:c607ec57:::smithi10428818-30:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:47.716 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:46.811436+0000", 2020-03-09T00:24:47.716 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.89603980100000002, 2020-03-09T00:24:47.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001842944, 2020-03-09T00:24:47.716 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:47.717 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:47.717 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:47.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:47.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811436+0000", 2020-03-09T00:24:47.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811436+0000", 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7788999999999998e-05 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811494+0000", 2020-03-09T00:24:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3485999999999999e-05 2020-03-09T00:24:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811508+0000", 2020-03-09T00:24:47.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6356000000000001e-05 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811524+0000", 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5699999999999998e-06 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:47.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.811531+0000", 2020-03-09T00:24:47.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014290559999999999 2020-03-09T00:24:47.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:47.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812960+0000", 2020-03-09T00:24:47.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031968699999999998 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.813279+0000", 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001842944 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:47.722 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:47.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.723 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.3s3 37/32 ECSubRead(tid=35, to_read={3:c607ec57:::smithi10428818-30:head=0,122880,0}, subchunks={3:c607ec57:::smithi10428818-30:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:47.723 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:46.812432+0000", 2020-03-09T00:24:47.723 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.89504409799999995, 2020-03-09T00:24:47.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010695030000000001, 2020-03-09T00:24:47.723 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:47.724 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:47.724 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:47.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:47.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812432+0000", 2020-03-09T00:24:47.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:47.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:47.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812432+0000", 2020-03-09T00:24:47.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1748000000000005e-05 2020-03-09T00:24:47.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:47.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812504+0000", 2020-03-09T00:24:47.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4555999999999999e-05 2020-03-09T00:24:47.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:47.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812518+0000", 2020-03-09T00:24:47.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1529999999999999e-05 2020-03-09T00:24:47.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:47.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812540+0000", 2020-03-09T00:24:47.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1310000000000006e-06 2020-03-09T00:24:47.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:47.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.812548+0000", 2020-03-09T00:24:47.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00077424500000000001 2020-03-09T00:24:47.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:47.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.813322+0000", 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017929299999999999 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:46.813502+0000", 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010695030000000001 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:47.729 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:47.730 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:47.730 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:47.730 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:47.730 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:47.731 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:24:47.757 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:47.745+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:47.812 INFO:tasks.rados.rados.0.smithi104.stdout:60: done (3 left) 2020-03-09T00:24:47.812 INFO:tasks.rados.rados.0.smithi104.stdout:61: done (2 left) 2020-03-09T00:24:47.812 INFO:tasks.rados.rados.0.smithi104.stdout:62: done (1 left) 2020-03-09T00:24:47.812 INFO:tasks.rados.rados.0.smithi104.stdout:63: done (0 left) 2020-03-09T00:24:47.812 INFO:tasks.rados.rados.0.smithi104.stdout:64: read oid 39 snap -1 2020-03-09T00:24:47.813 INFO:tasks.rados.rados.0.smithi104.stdout:64: expect (ObjNum 38 snap 0 seq_num 38) 2020-03-09T00:24:47.813 INFO:tasks.rados.rados.0.smithi104.stdout:65: read oid 47 snap 1 2020-03-09T00:24:47.813 INFO:tasks.rados.rados.0.smithi104.stdout:65: expect (ObjNum 46 snap 0 seq_num 46) 2020-03-09T00:24:47.813 INFO:tasks.rados.rados.0.smithi104.stdout:66: rollback oid 13 current snap is 2 2020-03-09T00:24:47.813 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 13 to 1 2020-03-09T00:24:47.813 INFO:tasks.rados.rados.0.smithi104.stdout:67: copy_from oid 36 from oid 12 current snap is 2 2020-03-09T00:24:47.813 INFO:tasks.rados.rados.0.smithi104.stdout:68: rollback oid 27 current snap is 2 2020-03-09T00:24:47.814 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 27 to 1 2020-03-09T00:24:47.814 INFO:tasks.rados.rados.0.smithi104.stdout:69: delete oid 7 current snap is 2 2020-03-09T00:24:47.815 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-03-09T00:24:47.822 INFO:tasks.rados.rados.0.smithi104.stdout:68: finishing rollback tid 0 to smithi10428818-27 2020-03-09T00:24:47.822 INFO:tasks.rados.rados.0.smithi104.stdout:69: done (5 left) 2020-03-09T00:24:47.822 INFO:tasks.rados.rados.0.smithi104.stdout:70: rmattr oid 41 current snap is 2 2020-03-09T00:24:47.822 INFO:tasks.rados.rados.0.smithi104.stdout:70: done (5 left) 2020-03-09T00:24:47.822 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 8 current snap is 2 2020-03-09T00:24:47.823 INFO:tasks.rados.rados.0.smithi104.stdout:71: seq_num 54 ranges {131072=540672,671744=376832} 2020-03-09T00:24:47.826 INFO:tasks.rados.rados.0.smithi104.stdout:71: writing smithi10428818-8 from 131072 to 671744 tid 1 2020-03-09T00:24:47.829 INFO:tasks.rados.rados.0.smithi104.stdout:71: writing smithi10428818-8 from 671744 to 1048576 tid 2 2020-03-09T00:24:47.829 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 31 current snap is 2 2020-03-09T00:24:47.829 INFO:tasks.rados.rados.0.smithi104.stdout:72: seq_num 55 ranges {507904=557056,1064960=376832} 2020-03-09T00:24:47.835 INFO:tasks.rados.rados.0.smithi104.stdout:72: writing smithi10428818-31 from 507904 to 1064960 tid 1 2020-03-09T00:24:47.838 INFO:tasks.rados.rados.0.smithi104.stdout:72: writing smithi10428818-31 from 1064960 to 1441792 tid 2 2020-03-09T00:24:47.838 INFO:tasks.rados.rados.0.smithi104.stdout:73: rollback oid 50 current snap is 2 2020-03-09T00:24:47.838 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 50 to 1 2020-03-09T00:24:47.838 INFO:tasks.rados.rados.0.smithi104.stdout:64: expect (ObjNum 38 snap 0 seq_num 38) 2020-03-09T00:24:47.848 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:47.845+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:47.857 INFO:tasks.rados.rados.0.smithi104.stdout:68: finishing rollback tid 1 to smithi10428818-27 2020-03-09T00:24:47.857 INFO:tasks.rados.rados.0.smithi104.stdout:68: finishing rollback tid 2 to smithi10428818-27 2020-03-09T00:24:47.858 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 27 v 13 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-03-09T00:24:47.858 INFO:tasks.rados.rados.0.smithi104.stdout:66: finishing rollback tid 0 to smithi10428818-13 2020-03-09T00:24:47.858 INFO:tasks.rados.rados.0.smithi104.stdout:66: finishing rollback tid 1 to smithi10428818-13 2020-03-09T00:24:47.858 INFO:tasks.rados.rados.0.smithi104.stdout:66: finishing rollback tid 2 to smithi10428818-13 2020-03-09T00:24:47.858 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 13 v 14 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-03-09T00:24:47.858 INFO:tasks.rados.rados.0.smithi104.stdout:67: finishing copy_from to smithi10428818-36 2020-03-09T00:24:47.858 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 36 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-03-09T00:24:47.859 INFO:tasks.rados.rados.0.smithi104.stdout:67: finishing copy_from racing read to smithi10428818-36 2020-03-09T00:24:47.859 INFO:tasks.rados.rados.0.smithi104.stdout:64: done (7 left) 2020-03-09T00:24:47.859 INFO:tasks.rados.rados.0.smithi104.stdout:66: done (6 left) 2020-03-09T00:24:47.859 INFO:tasks.rados.rados.0.smithi104.stdout:67: done (5 left) 2020-03-09T00:24:47.859 INFO:tasks.rados.rados.0.smithi104.stdout:68: done (4 left) 2020-03-09T00:24:47.859 INFO:tasks.rados.rados.0.smithi104.stdout:74: copy_from oid 28 from oid 43 current snap is 2 2020-03-09T00:24:47.860 INFO:tasks.rados.rados.0.smithi104.stdout:65: expect (ObjNum 46 snap 0 seq_num 46) 2020-03-09T00:24:47.871 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:47.871 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:47.871 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:24:47.871 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:47.879 INFO:tasks.rados.rados.0.smithi104.stdout:71: finishing write tid 1 to smithi10428818-8 2020-03-09T00:24:47.880 INFO:tasks.rados.rados.0.smithi104.stdout:71: finishing write tid 2 to smithi10428818-8 2020-03-09T00:24:47.880 INFO:tasks.rados.rados.0.smithi104.stdout:71: finishing write tid 4 to smithi10428818-8 2020-03-09T00:24:47.880 INFO:tasks.rados.rados.0.smithi104.stdout:71: finishing write tid 5 to smithi10428818-8 2020-03-09T00:24:47.880 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 8 v 8 (ObjNum 54 snap 2 seq_num 54) dirty exists 2020-03-09T00:24:47.880 INFO:tasks.rados.rados.0.smithi104.stdout:71: left oid 8 (ObjNum 54 snap 2 seq_num 54) 2020-03-09T00:24:47.881 INFO:tasks.rados.rados.0.smithi104.stdout:73: finishing rollback tid 0 to smithi10428818-50 2020-03-09T00:24:47.881 INFO:tasks.rados.rados.0.smithi104.stdout:65: done (4 left) 2020-03-09T00:24:47.881 INFO:tasks.rados.rados.0.smithi104.stdout:71: done (3 left) 2020-03-09T00:24:47.881 INFO:tasks.rados.rados.0.smithi104.stdout:75: snap_create 2020-03-09T00:24:47.881 INFO:tasks.rados.rados.0.smithi104.stdout:73: finishing rollback tid 1 to smithi10428818-50 2020-03-09T00:24:47.881 INFO:tasks.rados.rados.0.smithi104.stdout:73: finishing rollback tid 2 to smithi10428818-50 2020-03-09T00:24:47.881 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 50 v 13 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-03-09T00:24:47.882 INFO:tasks.rados.rados.0.smithi104.stdout:72: finishing write tid 1 to smithi10428818-31 2020-03-09T00:24:47.882 INFO:tasks.rados.rados.0.smithi104.stdout:72: finishing write tid 2 to smithi10428818-31 2020-03-09T00:24:47.882 INFO:tasks.rados.rados.0.smithi104.stdout:72: finishing write tid 4 to smithi10428818-31 2020-03-09T00:24:47.882 INFO:tasks.rados.rados.0.smithi104.stdout:72: finishing write tid 5 to smithi10428818-31 2020-03-09T00:24:47.882 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 31 v 10 (ObjNum 55 snap 2 seq_num 55) dirty exists 2020-03-09T00:24:47.882 INFO:tasks.rados.rados.0.smithi104.stdout:72: left oid 31 (ObjNum 55 snap 2 seq_num 55) 2020-03-09T00:24:47.882 INFO:tasks.rados.rados.0.smithi104.stdout:74: finishing copy_from to smithi10428818-28 2020-03-09T00:24:47.883 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 28 v 5 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-03-09T00:24:47.883 INFO:tasks.rados.rados.0.smithi104.stdout:74: finishing copy_from racing read to smithi10428818-28 2020-03-09T00:24:47.884 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:24:47.948 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:47.947+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:47.998 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:47.999 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:47.999 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:24:47.999 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:24:47.999 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:48.007 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:24:48.050 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:48.049+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:48.125 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:48.125 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:24:48.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:24:48.126 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:24:48.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.126 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:177 3.7s0 3:e7e20030:::smithi10428818-9:head [read 0~147456,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:24:48.126 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.810699+0000", 2020-03-09T00:24:48.126 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3125436029999999, 2020-03-09T00:24:48.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0071747520000000004, 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 177 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.127 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810699+0000", 2020-03-09T00:24:48.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810699+0000", 2020-03-09T00:24:48.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9414999999999997e-05 2020-03-09T00:24:48.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810788+0000", 2020-03-09T00:24:48.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0033e-05 2020-03-09T00:24:48.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810798+0000", 2020-03-09T00:24:48.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5071e-05 2020-03-09T00:24:48.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.131 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.131 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810814+0000", 2020-03-09T00:24:48.131 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2150000000000004e-06 2020-03-09T00:24:48.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810821+0000", 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4592000000000002e-05 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.132 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810875+0000", 2020-03-09T00:24:48.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.956e-06 2020-03-09T00:24:48.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2020-03-09T00:24:48.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810882+0000", 2020-03-09T00:24:48.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0021210510000000001 2020-03-09T00:24:48.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813003+0000", 2020-03-09T00:24:48.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000125046 2020-03-09T00:24:48.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813128+0000", 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047453729999999998 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817874+0000", 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0071747520000000004 2020-03-09T00:24:48.135 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:178 3.7s0 3:e7e20030:::smithi10428818-9:head [read 0~147456,checksum] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.810881+0000", 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3123614859999999, 2020-03-09T00:24:48.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007400078, 2020-03-09T00:24:48.137 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.137 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:48.137 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:48.137 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:48.137 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:48.137 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 178 2020-03-09T00:24:48.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810881+0000", 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.139 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810881+0000", 2020-03-09T00:24:48.139 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.5536000000000003e-05 2020-03-09T00:24:48.139 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.139 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.139 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.139 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810957+0000", 2020-03-09T00:24:48.139 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0283e-05 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810967+0000", 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2204999999999999e-05 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.140 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810979+0000", 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6770000000000003e-06 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810984+0000", 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9762999999999998e-05 2020-03-09T00:24:48.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.142 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.142 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811034+0000", 2020-03-09T00:24:48.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1699999999999997e-06 2020-03-09T00:24:48.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.142 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:48.143 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811042+0000", 2020-03-09T00:24:48.143 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026799689999999999 2020-03-09T00:24:48.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.143 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813722+0000", 2020-03-09T00:24:48.143 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011045800000000001 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813832+0000", 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0044490170000000004 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.144 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818281+0000", 2020-03-09T00:24:48.145 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007400078 2020-03-09T00:24:48.145 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.145 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.145 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.147 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:179 3.7s0 3:e7e20030:::smithi10428818-9:head [read 0~147456,checksum] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:24:48.147 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.811037+0000", 2020-03-09T00:24:48.148 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.312205944, 2020-03-09T00:24:48.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076295599999999996, 2020-03-09T00:24:48.148 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.148 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:48.148 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:48.148 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:48.148 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:48.149 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 179 2020-03-09T00:24:48.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.149 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.149 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.149 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811037+0000", 2020-03-09T00:24:48.149 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.150 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811037+0000", 2020-03-09T00:24:48.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1561000000000001e-05 2020-03-09T00:24:48.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811108+0000", 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1589999999999998e-06 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811114+0000", 2020-03-09T00:24:48.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3157e-05 2020-03-09T00:24:48.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.152 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.152 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811128+0000", 2020-03-09T00:24:48.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6450000000000004e-06 2020-03-09T00:24:48.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811132+0000", 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6229e-05 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811188+0000", 2020-03-09T00:24:48.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1640000000000001e-06 2020-03-09T00:24:48.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.154 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:48.154 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811197+0000", 2020-03-09T00:24:48.154 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039656980000000001 2020-03-09T00:24:48.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815162+0000", 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018545500000000001 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815348+0000", 2020-03-09T00:24:48.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033184920000000001 2020-03-09T00:24:48.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.156 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818666+0000", 2020-03-09T00:24:48.156 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076295599999999996 2020-03-09T00:24:48.156 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.156 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.7s0 37/32 ECSubRead(tid=29, to_read={3:e7e20030:::smithi10428818-9:head=0,36864,0}, subchunks={3:e7e20030:::smithi10428818-9:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.813661+0000", 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3095814619999999, 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026753049999999998, 2020-03-09T00:24:48.157 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813661+0000", 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813661+0000", 2020-03-09T00:24:48.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813661+0000", 2020-03-09T00:24:48.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813661+0000", 2020-03-09T00:24:48.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6075000000000001e-05 2020-03-09T00:24:48.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813677+0000", 2020-03-09T00:24:48.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.5449999999999998e-06 2020-03-09T00:24:48.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.813685+0000", 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002226923 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815912+0000", 2020-03-09T00:24:48.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00042476200000000002 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816336+0000", 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026753049999999998 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.7s0 36/32 ECSubReadReply(tid=29, attrs_read=0))", 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.814566+0000", 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.308676339, 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001842532, 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.164 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814566+0000", 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814566+0000", 2020-03-09T00:24:48.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9102000000000003e-05 2020-03-09T00:24:48.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814645+0000", 2020-03-09T00:24:48.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1161999999999999e-05 2020-03-09T00:24:48.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.167 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.167 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814657+0000", 2020-03-09T00:24:48.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3569e-05 2020-03-09T00:24:48.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814670+0000", 2020-03-09T00:24:48.168 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4864e-05 2020-03-09T00:24:48.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814685+0000", 2020-03-09T00:24:48.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016824839999999999 2020-03-09T00:24:48.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816367+0000", 2020-03-09T00:24:48.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1350999999999998e-05 2020-03-09T00:24:48.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816409+0000", 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001842532 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.171 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.7s0 37/32 ECSubRead(tid=30, to_read={3:e7e20030:::smithi10428818-9:head=0,36864,0}, subchunks={3:e7e20030:::smithi10428818-9:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:48.171 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.814783+0000", 2020-03-09T00:24:48.171 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3084597840000001, 2020-03-09T00:24:48.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001769084, 2020-03-09T00:24:48.171 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.171 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814783+0000", 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814783+0000", 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814783+0000", 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814783+0000", 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5746999999999999e-05 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814809+0000", 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3730000000000004e-06 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814816+0000", 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001622851 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816439+0000", 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113113 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816552+0000", 2020-03-09T00:24:48.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001769084 2020-03-09T00:24:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.7s0 36/32 ECSubReadReply(tid=30, attrs_read=0))", 2020-03-09T00:24:48.177 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.815240+0000", 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.308002809, 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013788050000000001, 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815240+0000", 2020-03-09T00:24:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815240+0000", 2020-03-09T00:24:48.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000104379 2020-03-09T00:24:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815344+0000", 2020-03-09T00:24:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0609e-05 2020-03-09T00:24:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815355+0000", 2020-03-09T00:24:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7286e-05 2020-03-09T00:24:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815372+0000", 2020-03-09T00:24:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7779999999999998e-06 2020-03-09T00:24:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815377+0000", 2020-03-09T00:24:48.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012001760000000001 2020-03-09T00:24:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816577+0000", 2020-03-09T00:24:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1576999999999999e-05 2020-03-09T00:24:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816619+0000", 2020-03-09T00:24:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013788050000000001 2020-03-09T00:24:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.185 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.7s0 37/32 ECSubRead(tid=31, to_read={3:e7e20030:::smithi10428818-9:head=0,36864,0}, subchunks={3:e7e20030:::smithi10428818-9:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:48.185 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:48.185 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3074198079999999, 2020-03-09T00:24:48.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017299629999999999, 2020-03-09T00:24:48.185 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.185 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.186 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:48.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:48.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:48.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:48.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010458900000000001 2020-03-09T00:24:48.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816869+0000", 2020-03-09T00:24:48.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0730000000000005e-06 2020-03-09T00:24:48.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816877+0000", 2020-03-09T00:24:48.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00055366599999999997 2020-03-09T00:24:48.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817430+0000", 2020-03-09T00:24:48.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012233400000000001 2020-03-09T00:24:48.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817553+0000", 2020-03-09T00:24:48.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017299629999999999 2020-03-09T00:24:48.192 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.192 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.192 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.193 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/32 ECSubReadReply(tid=29, attrs_read=0))", 2020-03-09T00:24:48.193 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.816331+0000", 2020-03-09T00:24:48.193 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3069111600000001, 2020-03-09T00:24:48.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001567768, 2020-03-09T00:24:48.193 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.193 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.193 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816331+0000", 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816331+0000", 2020-03-09T00:24:48.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816331+0000", 2020-03-09T00:24:48.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816331+0000", 2020-03-09T00:24:48.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00055977200000000005 2020-03-09T00:24:48.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816891+0000", 2020-03-09T00:24:48.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5720000000000002e-06 2020-03-09T00:24:48.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816897+0000", 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00067877699999999998 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817576+0000", 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00032364699999999999 2020-03-09T00:24:48.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817899+0000", 2020-03-09T00:24:48.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001567768 2020-03-09T00:24:48.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.199 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.200 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/32 ECSubReadReply(tid=30, attrs_read=0))", 2020-03-09T00:24:48.200 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.816550+0000", 2020-03-09T00:24:48.200 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3066924639999999, 2020-03-09T00:24:48.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001443943, 2020-03-09T00:24:48.201 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.201 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.201 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816550+0000", 2020-03-09T00:24:48.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816550+0000", 2020-03-09T00:24:48.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816550+0000", 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816550+0000", 2020-03-09T00:24:48.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00035884600000000001 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816909+0000", 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9510000000000002e-06 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816912+0000", 2020-03-09T00:24:48.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001024935 2020-03-09T00:24:48.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817937+0000", 2020-03-09T00:24:48.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7210999999999998e-05 2020-03-09T00:24:48.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817994+0000", 2020-03-09T00:24:48.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001443943 2020-03-09T00:24:48.206 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.207 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.207 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.207 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.7s0 35/32 ECSubReadReply(tid=30, attrs_read=0))", 2020-03-09T00:24:48.207 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.816881+0000", 2020-03-09T00:24:48.207 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3063618459999999, 2020-03-09T00:24:48.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014107709999999999, 2020-03-09T00:24:48.208 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.208 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.208 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816881+0000", 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816881+0000", 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6966999999999995e-05 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816978+0000", 2020-03-09T00:24:48.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3379999999999994e-06 2020-03-09T00:24:48.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816987+0000", 2020-03-09T00:24:48.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.224e-05 2020-03-09T00:24:48.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816999+0000", 2020-03-09T00:24:48.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8810000000000004e-06 2020-03-09T00:24:48.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817004+0000", 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010267779999999999 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818031+0000", 2020-03-09T00:24:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026056700000000001 2020-03-09T00:24:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818292+0000", 2020-03-09T00:24:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014107709999999999 2020-03-09T00:24:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.214 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.214 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.214 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/32 ECSubWrite(tid=42, reqid=client.4751.0:192, at_version=38'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:48.215 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.817181+0000", 2020-03-09T00:24:48.216 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.30606181700000001, 2020-03-09T00:24:48.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023822320000000002, 2020-03-09T00:24:48.216 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.216 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.216 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817181+0000", 2020-03-09T00:24:48.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817181+0000", 2020-03-09T00:24:48.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7033000000000001e-05 2020-03-09T00:24:48.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817238+0000", 2020-03-09T00:24:48.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1949999999999997e-05 2020-03-09T00:24:48.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817270+0000", 2020-03-09T00:24:48.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4667999999999999e-05 2020-03-09T00:24:48.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817284+0000", 2020-03-09T00:24:48.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8590000000000005e-06 2020-03-09T00:24:48.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817292+0000", 2020-03-09T00:24:48.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000781688 2020-03-09T00:24:48.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818074+0000", 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2717999999999999e-05 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818087+0000", 2020-03-09T00:24:48.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001428131 2020-03-09T00:24:48.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819515+0000", 2020-03-09T00:24:48.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8185e-05 2020-03-09T00:24:48.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819563+0000", 2020-03-09T00:24:48.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023822320000000002 2020-03-09T00:24:48.223 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.223 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.223 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.224 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s2 38/32 ECSubWrite(tid=20, reqid=client.4751.0:190, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:24:48.224 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.823370+0000", 2020-03-09T00:24:48.224 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.299872519, 2020-03-09T00:24:48.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027779860000000001, 2020-03-09T00:24:48.224 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.225 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.225 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.225 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823370+0000", 2020-03-09T00:24:48.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823370+0000", 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9339000000000006e-05 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823459+0000", 2020-03-09T00:24:48.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.889e-05 2020-03-09T00:24:48.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823478+0000", 2020-03-09T00:24:48.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7156999999999999e-05 2020-03-09T00:24:48.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823495+0000", 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5189999999999998e-06 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823502+0000", 2020-03-09T00:24:48.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027186700000000002 2020-03-09T00:24:48.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823774+0000", 2020-03-09T00:24:48.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4321000000000001e-05 2020-03-09T00:24:48.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823788+0000", 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023064829999999998 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826095+0000", 2020-03-09T00:24:48.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3409999999999999e-05 2020-03-09T00:24:48.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826148+0000", 2020-03-09T00:24:48.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027779860000000001 2020-03-09T00:24:48.231 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 38/32 ECSubWrite(tid=45, reqid=client.4751.0:194, at_version=0'0, trim_to=0'0, roll_forward_to=38'13))", 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.826216+0000", 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.297026964, 2020-03-09T00:24:48.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001775712, 2020-03-09T00:24:48.233 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.233 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.233 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826216+0000", 2020-03-09T00:24:48.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826216+0000", 2020-03-09T00:24:48.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3314999999999999e-05 2020-03-09T00:24:48.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826279+0000", 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4663e-05 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826294+0000", 2020-03-09T00:24:48.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.959e-05 2020-03-09T00:24:48.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.236 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.236 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826313+0000", 2020-03-09T00:24:48.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.039e-05 2020-03-09T00:24:48.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826324+0000", 2020-03-09T00:24:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4966999999999998e-05 2020-03-09T00:24:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826369+0000", 2020-03-09T00:24:48.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6463000000000001e-05 2020-03-09T00:24:48.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826385+0000", 2020-03-09T00:24:48.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00054624600000000001 2020-03-09T00:24:48.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826931+0000", 2020-03-09T00:24:48.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010600780000000001 2020-03-09T00:24:48.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827991+0000", 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001775712 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.240 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.5s2 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:48.241 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.827763+0000", 2020-03-09T00:24:48.241 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.29547927600000001, 2020-03-09T00:24:48.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0085507429999999995, 2020-03-09T00:24:48.241 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.241 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.241 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827763+0000", 2020-03-09T00:24:48.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827763+0000", 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000278683 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828042+0000", 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4984e-05 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828067+0000", 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5858e-05 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828083+0000", 2020-03-09T00:24:48.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.345e-06 2020-03-09T00:24:48.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828090+0000", 2020-03-09T00:24:48.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113941 2020-03-09T00:24:48.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828204+0000", 2020-03-09T00:24:48.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3959e-05 2020-03-09T00:24:48.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828218+0000", 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0080567880000000005 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836275+0000", 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9184999999999998e-05 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.248 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836314+0000", 2020-03-09T00:24:48.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0085507429999999995 2020-03-09T00:24:48.248 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.248 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.248 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.5s2 38/32 ECSubWrite(tid=11, reqid=client.4751.0:197, at_version=38'7, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.833352+0000", 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.289890176, 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003149177, 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.249 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833352+0000", 2020-03-09T00:24:48.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833352+0000", 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000142368 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833495+0000", 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2558999999999999e-05 2020-03-09T00:24:48.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833517+0000", 2020-03-09T00:24:48.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7433999999999999e-05 2020-03-09T00:24:48.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833535+0000", 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0260000000000002e-06 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833542+0000", 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8977999999999997e-05 2020-03-09T00:24:48.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833601+0000", 2020-03-09T00:24:48.254 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8077e-05 2020-03-09T00:24:48.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.833619+0000", 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0028552809999999999 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836474+0000", 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.7453999999999998e-05 2020-03-09T00:24:48.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.256 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.256 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836502+0000", 2020-03-09T00:24:48.256 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003149177 2020-03-09T00:24:48.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.256 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.5s2 38/32 ECSubWrite(tid=12, reqid=client.4751.0:198, at_version=38'8, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.836414+0000", 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.28682877099999998, 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026101919999999999, 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.257 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836414+0000", 2020-03-09T00:24:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836414+0000", 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.978e-05 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836474+0000", 2020-03-09T00:24:48.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0169999999999998e-05 2020-03-09T00:24:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836494+0000", 2020-03-09T00:24:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.562e-05 2020-03-09T00:24:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836509+0000", 2020-03-09T00:24:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8210000000000004e-06 2020-03-09T00:24:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836516+0000", 2020-03-09T00:24:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3015000000000001e-05 2020-03-09T00:24:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836559+0000", 2020-03-09T00:24:48.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1941e-05 2020-03-09T00:24:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836571+0000", 2020-03-09T00:24:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002420251 2020-03-09T00:24:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.838991+0000", 2020-03-09T00:24:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2594e-05 2020-03-09T00:24:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.839024+0000", 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026101919999999999 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s4 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:48.265 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.837641+0000", 2020-03-09T00:24:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.285601196, 2020-03-09T00:24:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086515399999999992, 2020-03-09T00:24:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.266 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837641+0000", 2020-03-09T00:24:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837641+0000", 2020-03-09T00:24:48.267 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207486 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837849+0000", 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0214999999999999e-05 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.268 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837869+0000", 2020-03-09T00:24:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2768999999999999e-05 2020-03-09T00:24:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837882+0000", 2020-03-09T00:24:48.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0450000000000003e-06 2020-03-09T00:24:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837889+0000", 2020-03-09T00:24:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0652000000000002e-05 2020-03-09T00:24:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837940+0000", 2020-03-09T00:24:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7520000000000008e-06 2020-03-09T00:24:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837949+0000", 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0082991139999999998 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.846248+0000", 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4506999999999999e-05 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.846293+0000", 2020-03-09T00:24:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086515399999999992 2020-03-09T00:24:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s4 38/32 ECSubWrite(tid=16, reqid=client.4751.0:201, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.844167+0000", 2020-03-09T00:24:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.27907557399999999, 2020-03-09T00:24:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002314857, 2020-03-09T00:24:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.274 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844167+0000", 2020-03-09T00:24:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844167+0000", 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4005000000000005e-05 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844261+0000", 2020-03-09T00:24:48.276 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3152e-05 2020-03-09T00:24:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844274+0000", 2020-03-09T00:24:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1041e-05 2020-03-09T00:24:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844285+0000", 2020-03-09T00:24:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.682e-06 2020-03-09T00:24:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.278 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844291+0000", 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7256000000000003e-05 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844338+0000", 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3186999999999999e-05 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844351+0000", 2020-03-09T00:24:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020912909999999999 2020-03-09T00:24:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.280 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.846443+0000", 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9242999999999997e-05 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.846482+0000", 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002314857 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.281 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s4 38/32 ECSubWrite(tid=17, reqid=client.4751.0:202, at_version=38'10, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.847143+0000", 2020-03-09T00:24:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.27609989099999999, 2020-03-09T00:24:48.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001856431, 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847143+0000", 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:48.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847143+0000", 2020-03-09T00:24:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0026999999999999e-05 2020-03-09T00:24:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847173+0000", 2020-03-09T00:24:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3071000000000001e-05 2020-03-09T00:24:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847186+0000", 2020-03-09T00:24:48.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0010000000000007e-06 2020-03-09T00:24:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847195+0000", 2020-03-09T00:24:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6800000000000001e-06 2020-03-09T00:24:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847199+0000", 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.434e-05 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847234+0000", 2020-03-09T00:24:48.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9940000000000003e-06 2020-03-09T00:24:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.847241+0000", 2020-03-09T00:24:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001727297 2020-03-09T00:24:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.848968+0000", 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1021e-05 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.848999+0000", 2020-03-09T00:24:48.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001856431 2020-03-09T00:24:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:48.290 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:48.290 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:48.292 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:48.290+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:48.292 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:24:48.392 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:48.389+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:48.429 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:48.429 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:48.429 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:24:48.429 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:48.438 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:24:48.493 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:48.492+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:48.562 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:48.562 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:48.563 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:24:48.563 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:24:48.563 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:48.571 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:24:48.594 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:48.593+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:48.695 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:48.693+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:48.696 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:24:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:24:48.696 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:24:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.es2 38/32 ECSubRead(tid=29, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.814659+0000", 2020-03-09T00:24:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87943904799999995, 2020-03-09T00:24:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010645520000000001, 2020-03-09T00:24:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.697 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814659+0000", 2020-03-09T00:24:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814659+0000", 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1724000000000003e-05 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814690+0000", 2020-03-09T00:24:48.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7020000000000007e-06 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814699+0000", 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3560999999999999e-05 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814713+0000", 2020-03-09T00:24:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7549999999999997e-06 2020-03-09T00:24:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814719+0000", 2020-03-09T00:24:48.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4870999999999995e-05 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814794+0000", 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092893900000000005 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815723+0000", 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010645520000000001 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.703 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.es2 38/32 ECSubRead(tid=30, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.815146+0000", 2020-03-09T00:24:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87895139300000003, 2020-03-09T00:24:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097355900000000003, 2020-03-09T00:24:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815146+0000", 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815146+0000", 2020-03-09T00:24:48.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6650999999999996e-05 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815213+0000", 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3145e-05 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815226+0000", 2020-03-09T00:24:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9264e-05 2020-03-09T00:24:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815255+0000", 2020-03-09T00:24:48.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4569999999999994e-06 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815265+0000", 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050765200000000004 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815773+0000", 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034738999999999999 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816120+0000", 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097355900000000003 2020-03-09T00:24:48.709 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s1 38/32 ECSubWrite(tid=46, reqid=client.4751.0:195, at_version=38'16, trim_to=0'0, roll_forward_to=35'14))", 2020-03-09T00:24:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.815244+0000", 2020-03-09T00:24:48.710 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87885378800000002, 2020-03-09T00:24:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014121209999999999, 2020-03-09T00:24:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815244+0000", 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815244+0000", 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9752000000000006e-05 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815314+0000", 2020-03-09T00:24:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4808999999999998e-05 2020-03-09T00:24:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815349+0000", 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7611000000000001e-05 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815366+0000", 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9470000000000003e-06 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815374+0000", 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6629000000000002e-05 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815441+0000", 2020-03-09T00:24:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3654e-05 2020-03-09T00:24:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815454+0000", 2020-03-09T00:24:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011364319999999999 2020-03-09T00:24:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816591+0000", 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5287e-05 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816656+0000", 2020-03-09T00:24:48.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014121209999999999 2020-03-09T00:24:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.es2 38/32 ECSubRead(tid=31, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:48.719 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.815713+0000", 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87838511900000005, 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079279199999999995, 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815713+0000", 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815713+0000", 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8885000000000003e-05 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815772+0000", 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3879e-05 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815795+0000", 2020-03-09T00:24:48.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9298999999999999e-05 2020-03-09T00:24:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815815+0000", 2020-03-09T00:24:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6899999999999992e-06 2020-03-09T00:24:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815822+0000", 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036611899999999998 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816189+0000", 2020-03-09T00:24:48.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031691999999999999 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816505+0000", 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079279199999999995 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 38/32 ECSubWrite(tid=32, reqid=client.4751.0:187, at_version=38'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.817644+0000", 2020-03-09T00:24:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.876454022, 2020-03-09T00:24:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099396899999999993, 2020-03-09T00:24:48.726 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817644+0000", 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817644+0000", 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4471999999999998e-05 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817678+0000", 2020-03-09T00:24:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8695e-05 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817697+0000", 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0003999999999999e-05 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817707+0000", 2020-03-09T00:24:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2769999999999996e-06 2020-03-09T00:24:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817712+0000", 2020-03-09T00:24:48.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7266999999999999e-05 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817769+0000", 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2880999999999999e-05 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817782+0000", 2020-03-09T00:24:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079631299999999999 2020-03-09T00:24:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818579+0000", 2020-03-09T00:24:48.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.906e-05 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818638+0000", 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099396899999999993 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.733 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 38/32 ECSubWrite(tid=33, reqid=client.4751.0:188, at_version=38'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.819093+0000", 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87500472500000004, 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093362000000000002, 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.734 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819093+0000", 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819093+0000", 2020-03-09T00:24:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0834999999999994e-05 2020-03-09T00:24:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819174+0000", 2020-03-09T00:24:48.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7943000000000001e-05 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819192+0000", 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0028e-05 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819202+0000", 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0448000000000001e-05 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819212+0000", 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3974999999999997e-05 2020-03-09T00:24:48.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819266+0000", 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1626e-05 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819278+0000", 2020-03-09T00:24:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000695022 2020-03-09T00:24:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819973+0000", 2020-03-09T00:24:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3742999999999997e-05 2020-03-09T00:24:48.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820027+0000", 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093362000000000002 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 38/32 ECSubWrite(tid=34, reqid=client.4751.0:189, at_version=38'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.820242+0000", 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87385533599999998, 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059170200000000005, 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.742 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820242+0000", 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820242+0000", 2020-03-09T00:24:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8907999999999998e-05 2020-03-09T00:24:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820311+0000", 2020-03-09T00:24:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3555000000000001e-05 2020-03-09T00:24:48.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820335+0000", 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3519999999999999e-05 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820348+0000", 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6850000000000004e-06 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820355+0000", 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2321000000000003e-05 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820407+0000", 2020-03-09T00:24:48.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3267e-05 2020-03-09T00:24:48.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820421+0000", 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037071499999999998 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820791+0000", 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2731000000000001e-05 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820834+0000", 2020-03-09T00:24:48.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059170200000000005 2020-03-09T00:24:48.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.749 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s1 38/32 ECSubWrite(tid=47, reqid=client.4751.0:195, at_version=0'0, trim_to=0'0, roll_forward_to=38'16))", 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.821650+0000", 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87244773200000003, 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049487399999999996, 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.750 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821650+0000", 2020-03-09T00:24:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821650+0000", 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3816000000000003e-05 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821684+0000", 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1807000000000001e-05 2020-03-09T00:24:48.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821696+0000", 2020-03-09T00:24:48.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2024e-05 2020-03-09T00:24:48.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821708+0000", 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9970000000000001e-06 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821715+0000", 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7797999999999999e-05 2020-03-09T00:24:48.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821762+0000", 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0920999999999999e-05 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.821773+0000", 2020-03-09T00:24:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033264200000000002 2020-03-09T00:24:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.822106+0000", 2020-03-09T00:24:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8868999999999997e-05 2020-03-09T00:24:48.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.822145+0000", 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049487399999999996 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.758 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s3 38/32 ECSubWrite(tid=20, reqid=client.4751.0:190, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:24:48.758 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.822896+0000", 2020-03-09T00:24:48.758 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87120153499999997, 2020-03-09T00:24:48.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018780649999999999, 2020-03-09T00:24:48.758 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.758 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.822896+0000", 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.822896+0000", 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0469999999999994e-05 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.822977+0000", 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8971000000000001e-05 2020-03-09T00:24:48.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823006+0000", 2020-03-09T00:24:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2317999999999999e-05 2020-03-09T00:24:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823028+0000", 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8400000000000007e-06 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823038+0000", 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0638999999999998e-05 2020-03-09T00:24:48.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823118+0000", 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5140000000000001e-06 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823127+0000", 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016098079999999999 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.824737+0000", 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7505000000000001e-05 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.824774+0000", 2020-03-09T00:24:48.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018780649999999999 2020-03-09T00:24:48.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.765 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 38/32 ECSubWrite(tid=35, reqid=client.4751.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.825631+0000", 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.86846677100000003, 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071963599999999995, 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.766 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825631+0000", 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825631+0000", 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1739999999999997e-05 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825673+0000", 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0339e-05 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825683+0000", 2020-03-09T00:24:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1858000000000001e-05 2020-03-09T00:24:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825695+0000", 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0550000000000002e-06 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825702+0000", 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1799999999999997e-05 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825774+0000", 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8532000000000001e-05 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825792+0000", 2020-03-09T00:24:48.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051851000000000002 2020-03-09T00:24:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826311+0000", 2020-03-09T00:24:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9802000000000003e-05 2020-03-09T00:24:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826351+0000", 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071963599999999995 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.774 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s3 38/32 ECSubWrite(tid=21, reqid=client.4751.0:190, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2020-03-09T00:24:48.774 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.826742+0000", 2020-03-09T00:24:48.774 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.86735589800000001, 2020-03-09T00:24:48.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075987400000000001, 2020-03-09T00:24:48.774 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.774 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.774 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826742+0000", 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826742+0000", 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3040999999999998e-05 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826775+0000", 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9510000000000001e-06 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826785+0000", 2020-03-09T00:24:48.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1416e-05 2020-03-09T00:24:48.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826796+0000", 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6260000000000001e-06 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826805+0000", 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5077000000000002e-05 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826850+0000", 2020-03-09T00:24:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6946999999999999e-05 2020-03-09T00:24:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826867+0000", 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00057627600000000002 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.827443+0000", 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8539999999999999e-05 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.781 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.781 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.827502+0000", 2020-03-09T00:24:48.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075987400000000001 2020-03-09T00:24:48.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.781 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.85671112999999999, 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048492969999999998, 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:24:48.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026746099999999998 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837654+0000", 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8997e-05 2020-03-09T00:24:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837683+0000", 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4249999999999999e-05 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837717+0000", 2020-03-09T00:24:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9910000000000002e-06 2020-03-09T00:24:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837725+0000", 2020-03-09T00:24:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013534699999999999 2020-03-09T00:24:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837861+0000", 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9768e-05 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837890+0000", 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042879290000000002 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842178+0000", 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7553999999999998e-05 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842236+0000", 2020-03-09T00:24:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048492969999999998 2020-03-09T00:24:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=16, reqid=client.4751.0:201, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.843013+0000", 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.85108486999999999, 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032784310000000001, 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843013+0000", 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843013+0000", 2020-03-09T00:24:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000138565 2020-03-09T00:24:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843151+0000", 2020-03-09T00:24:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2470999999999999e-05 2020-03-09T00:24:48.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843174+0000", 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0281000000000001e-05 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843194+0000", 2020-03-09T00:24:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2519999999999999e-06 2020-03-09T00:24:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843202+0000", 2020-03-09T00:24:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7967000000000003e-05 2020-03-09T00:24:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843260+0000", 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4036000000000001e-05 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843274+0000", 2020-03-09T00:24:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029609940000000002 2020-03-09T00:24:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846235+0000", 2020-03-09T00:24:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5865000000000001e-05 2020-03-09T00:24:48.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846291+0000", 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032784310000000001 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.798 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=17, reqid=client.4751.0:202, at_version=38'10, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:48.798 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.846400+0000", 2020-03-09T00:24:48.798 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.84769810599999995, 2020-03-09T00:24:48.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097257099999999998, 2020-03-09T00:24:48.798 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.798 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.799 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846400+0000", 2020-03-09T00:24:48.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846400+0000", 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8084000000000002e-05 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846458+0000", 2020-03-09T00:24:48.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0944000000000001e-05 2020-03-09T00:24:48.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846479+0000", 2020-03-09T00:24:48.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6935e-05 2020-03-09T00:24:48.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846496+0000", 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7050000000000002e-06 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846502+0000", 2020-03-09T00:24:48.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7809000000000002e-05 2020-03-09T00:24:48.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846560+0000", 2020-03-09T00:24:48.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2928999999999999e-05 2020-03-09T00:24:48.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846573+0000", 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073761900000000001 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847311+0000", 2020-03-09T00:24:48.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1545999999999997e-05 2020-03-09T00:24:48.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847372+0000", 2020-03-09T00:24:48.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097257099999999998 2020-03-09T00:24:48.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.805 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=18, reqid=client.4751.0:202, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.849508+0000", 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.84458996900000005, 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00048824600000000001, 2020-03-09T00:24:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849508+0000", 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849508+0000", 2020-03-09T00:24:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1238000000000001e-05 2020-03-09T00:24:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849539+0000", 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0547000000000001e-05 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849550+0000", 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9219999999999992e-06 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849560+0000", 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0209999999999999e-06 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849565+0000", 2020-03-09T00:24:48.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2185999999999998e-05 2020-03-09T00:24:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849617+0000", 2020-03-09T00:24:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2277e-05 2020-03-09T00:24:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849639+0000", 2020-03-09T00:24:48.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000305779 2020-03-09T00:24:48.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849945+0000", 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1276000000000002e-05 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849996+0000", 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00048824600000000001 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.813 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:24:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.864016+0000", 2020-03-09T00:24:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.83008156099999997, 2020-03-09T00:24:48.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044756739999999998, 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864016+0000", 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864016+0000", 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011600200000000001 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864132+0000", 2020-03-09T00:24:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8845999999999999e-05 2020-03-09T00:24:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864151+0000", 2020-03-09T00:24:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2444e-05 2020-03-09T00:24:48.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864164+0000", 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.989e-06 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864169+0000", 2020-03-09T00:24:48.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7182999999999998e-05 2020-03-09T00:24:48.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864226+0000", 2020-03-09T00:24:48.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.306e-05 2020-03-09T00:24:48.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864239+0000", 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042089520000000002 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.868448+0000", 2020-03-09T00:24:48.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4197999999999999e-05 2020-03-09T00:24:48.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.868492+0000", 2020-03-09T00:24:48.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044756739999999998 2020-03-09T00:24:48.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 38/32 ECSubWrite(tid=6, reqid=client.4751.0:207, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.870612+0000", 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.82348628499999998, 2020-03-09T00:24:48.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073473600000000005, 2020-03-09T00:24:48.823 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:48.823 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:48.823 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:48.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:48.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870612+0000", 2020-03-09T00:24:48.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870612+0000", 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015101599999999999 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:48.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870763+0000", 2020-03-09T00:24:48.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8924e-05 2020-03-09T00:24:48.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:48.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870781+0000", 2020-03-09T00:24:48.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1274e-05 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870803+0000", 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0411000000000001e-05 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870813+0000", 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6025000000000006e-05 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870879+0000", 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3302000000000001e-05 2020-03-09T00:24:48.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:48.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870892+0000", 2020-03-09T00:24:48.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041476600000000001 2020-03-09T00:24:48.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.871307+0000", 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9017999999999998e-05 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.871346+0000", 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073473600000000005 2020-03-09T00:24:48.829 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:48.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:48.830 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:48.831 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:24:48.831 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:24:48.831 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2020-03-09T00:24:48.831 INFO:tasks.ceph.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 16 2020-03-09T00:24:48.831 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-03-09T00:24:48.834 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is unset 2020-03-09T00:24:48.834 INFO:tasks.rados.rados.0.smithi104.stdout:72: done (3 left) 2020-03-09T00:24:48.834 INFO:tasks.rados.rados.0.smithi104.stdout:73: done (2 left) 2020-03-09T00:24:48.834 INFO:tasks.rados.rados.0.smithi104.stdout:74: done (1 left) 2020-03-09T00:24:48.834 INFO:tasks.rados.rados.0.smithi104.stdout:75: done (0 left) 2020-03-09T00:24:48.835 INFO:tasks.rados.rados.0.smithi104.stdout:76: rmattr oid 40 current snap is 3 2020-03-09T00:24:48.835 INFO:tasks.rados.rados.0.smithi104.stdout:76: done (0 left) 2020-03-09T00:24:48.835 INFO:tasks.rados.rados.0.smithi104.stdout:77: copy_from oid 46 from oid 39 current snap is 3 2020-03-09T00:24:48.835 INFO:tasks.rados.rados.0.smithi104.stdout:78: snap_create 2020-03-09T00:24:48.835 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:48.830+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:48.836 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:24:48.836 INFO:tasks.rados.rados.0.smithi104.stdout:77: finishing copy_from to smithi10428818-46 2020-03-09T00:24:48.836 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 46 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-03-09T00:24:48.837 INFO:tasks.rados.rados.0.smithi104.stdout:77: finishing copy_from racing read to smithi10428818-46 2020-03-09T00:24:48.846 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:24:48.932 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:48.931+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:24:49.001 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:49.001 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:49.001 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:24:49.001 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:49.010 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:24:49.014 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:49.033 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:49.032+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:49.037 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:49.058 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:49.079 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:49.100 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:49.121 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:49.131 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:49.131 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:49.134 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:49.133+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:49.140 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:24:49.142 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:24:49.152 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2020-03-09T00:24:49.163 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2020-03-09T00:24:48.814655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20955136,"num_objects":51,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":127,"num_read_kb":5914,"num_write":111,"num_write_kb":21470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":16817784,"kb_used_data":40440,"kb_used_omap":0,"kb_used_meta":16777216,"kb_avail":1493131656,"statfs":{"total":1546188226560,"available":1528966815744,"internally_reserved":17179869184,"allocated":41410560,"data_stored":38691744,"data_compressed":364522,"data_compressed_allocated":794624,"data_compressed_original":1589248,"omap_allocated":0,"internal_metadata":17179869184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20955136,"num_objects":48,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":127,"num_read_kb":5914,"num_write":108,"num_write_kb":21456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019054"},"pg_stats":[{"pgid":"3.f","version":"35'9","reported_seq":"33","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.813363+0000","last_change":"2020-03-09T00:24:46.813943+0000","last_active":"2020-03-09T00:24:46.813363+0000","last_peered":"2020-03-09T00:24:46.813363+0000","last_clean":"2020-03-09T00:24:46.813363+0000","last_became_active":"2020-03-09T00:24:43.052414+0000","last_became_peered":"2020-03-09T00:24:43.052414+0000","last_unstale":"2020-03-09T00:24:46.813363+0000","last_undegraded":"2020-03-09T00:24:46.813363+0000","last_fullsized":"2020-03-09T00:24:46.813363+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2129920,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":9,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12,4,13,14,10,2],"acting":[11,12,4,13,14,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"35'10","reported_seq":"40","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.815347+0000","last_change":"2020-03-09T00:24:46.815861+0000","last_active":"2020-03-09T00:24:46.815347+0000","last_peered":"2020-03-09T00:24:46.815347+0000","last_clean":"2020-03-09T00:24:46.815347+0000","last_became_active":"2020-03-09T00:24:43.219818+0000","last_became_peered":"2020-03-09T00:24:43.219818+0000","last_unstale":"2020-03-09T00:24:46.815347+0000","last_undegraded":"2020-03-09T00:24:46.815347+0000","last_fullsized":"2020-03-09T00:24:46.815347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2457600,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":631,"num_write":10,"num_write_kb":2400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,11,2,0,3],"acting":[8,9,1,11,2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":"20","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.814233+0000","last_change":"2020-03-09T00:24:46.814690+0000","last_active":"2020-03-09T00:24:46.814233+0000","last_peered":"2020-03-09T00:24:46.814233+0000","last_clean":"2020-03-09T00:24:46.814233+0000","last_became_active":"2020-03-09T00:24:43.218239+0000","last_became_peered":"2020-03-09T00:24:43.218239+0000","last_unstale":"2020-03-09T00:24:46.814233+0000","last_undegraded":"2020-03-09T00:24:46.814233+0000","last_fullsized":"2020-03-09T00:24:46.814233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0,12,10,2,6],"acting":[7,3,0,12,10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"35'10","reported_seq":"35","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.811966+0000","last_change":"2020-03-09T00:24:46.812470+0000","last_active":"2020-03-09T00:24:46.811966+0000","last_peered":"2020-03-09T00:24:46.811966+0000","last_clean":"2020-03-09T00:24:46.811966+0000","last_became_active":"2020-03-09T00:24:43.052032+0000","last_became_peered":"2020-03-09T00:24:43.052032+0000","last_unstale":"2020-03-09T00:24:46.811966+0000","last_undegraded":"2020-03-09T00:24:46.811966+0000","last_fullsized":"2020-03-09T00:24:46.811966+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1900544,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":10,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,14,12,4,2,6],"acting":[11,10,14,12,4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"35'6","reported_seq":"25","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.951213+0000","last_change":"2020-03-09T00:24:43.087739+0000","last_active":"2020-03-09T00:24:44.951213+0000","last_peered":"2020-03-09T00:24:44.951213+0000","last_clean":"2020-03-09T00:24:44.951213+0000","last_became_active":"2020-03-09T00:24:43.086810+0000","last_became_peered":"2020-03-09T00:24:43.086810+0000","last_unstale":"2020-03-09T00:24:44.951213+0000","last_undegraded":"2020-03-09T00:24:44.951213+0000","last_fullsized":"2020-03-09T00:24:44.951213+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1163264,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1,9,8,7,14],"acting":[10,6,1,9,8,7,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.a","version":"35'2","reported_seq":"23","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.808989+0000","last_change":"2020-03-09T00:24:46.809452+0000","last_active":"2020-03-09T00:24:46.808989+0000","last_peered":"2020-03-09T00:24:46.808989+0000","last_clean":"2020-03-09T00:24:46.808989+0000","last_became_active":"2020-03-09T00:24:43.086461+0000","last_became_peered":"2020-03-09T00:24:43.086461+0000","last_unstale":"2020-03-09T00:24:46.808989+0000","last_undegraded":"2020-03-09T00:24:46.808989+0000","last_fullsized":"2020-03-09T00:24:46.808989+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":458752,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,8,15,11],"acting":[3,7,1,9,8,15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"35'14","reported_seq":"36","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.907206+0000","last_change":"2020-03-09T00:24:43.052560+0000","last_active":"2020-03-09T00:24:44.907206+0000","last_peered":"2020-03-09T00:24:44.907206+0000","last_clean":"2020-03-09T00:24:44.907206+0000","last_became_active":"2020-03-09T00:24:43.051454+0000","last_became_peered":"2020-03-09T00:24:43.051454+0000","last_unstale":"2020-03-09T00:24:44.907206+0000","last_undegraded":"2020-03-09T00:24:44.907206+0000","last_fullsized":"2020-03-09T00:24:44.907206+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2473984,"num_objects":7,"num_object_clones":0,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":14,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,14,4,15,10,6],"acting":[12,1,14,4,15,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"35'6","reported_seq":"32","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.818219+0000","last_change":"2020-03-09T00:24:46.818678+0000","last_active":"2020-03-09T00:24:46.818219+0000","last_peered":"2020-03-09T00:24:46.818219+0000","last_clean":"2020-03-09T00:24:46.818219+0000","last_became_active":"2020-03-09T00:24:43.218542+0000","last_became_peered":"2020-03-09T00:24:43.218542+0000","last_unstale":"2020-03-09T00:24:46.818219+0000","last_undegraded":"2020-03-09T00:24:46.818219+0000","last_fullsized":"2020-03-09T00:24:46.818219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1982464,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":2212,"num_write":6,"num_write_kb":1936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,0,4,14,15,1],"acting":[5,10,0,4,14,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":"17","reported_epoch":"36","state":"active+clean","last_fresh":"2020-03-09T00:24:46.159269+0000","last_change":"2020-03-09T00:24:28.135611+0000","last_active":"2020-03-09T00:24:46.159269+0000","last_peered":"2020-03-09T00:24:46.159269+0000","last_clean":"2020-03-09T00:24:46.159269+0000","last_became_active":"2020-03-09T00:24:28.135155+0000","last_became_peered":"2020-03-09T00:24:28.135155+0000","last_unstale":"2020-03-09T00:24:46.159269+0000","last_undegraded":"2020-03-09T00:24:46.159269+0000","last_fullsized":"2020-03-09T00:24:46.159269+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12],"acting":[6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"35'4","reported_seq":"26","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.814810+0000","last_change":"2020-03-09T00:24:46.815308+0000","last_active":"2020-03-09T00:24:46.814810+0000","last_peered":"2020-03-09T00:24:46.814810+0000","last_clean":"2020-03-09T00:24:46.814810+0000","last_became_active":"2020-03-09T00:24:42.800823+0000","last_became_peered":"2020-03-09T00:24:42.800823+0000","last_unstale":"2020-03-09T00:24:46.814810+0000","last_undegraded":"2020-03-09T00:24:46.814810+0000","last_fullsized":"2020-03-09T00:24:46.814810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":819200,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,14,13,7,11,8],"acting":[15,6,14,13,7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":"16","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.820745+0000","last_change":"2020-03-09T00:24:27.942443+0000","last_active":"2020-03-09T00:24:44.820745+0000","last_peered":"2020-03-09T00:24:44.820745+0000","last_clean":"2020-03-09T00:24:44.820745+0000","last_became_active":"2020-03-09T00:24:27.942050+0000","last_became_peered":"2020-03-09T00:24:27.942050+0000","last_unstale":"2020-03-09T00:24:44.820745+0000","last_undegraded":"2020-03-09T00:24:44.820745+0000","last_fullsized":"2020-03-09T00:24:44.820745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"35'9","reported_seq":"36","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.817187+0000","last_change":"2020-03-09T00:24:46.809593+0000","last_active":"2020-03-09T00:24:46.817187+0000","last_peered":"2020-03-09T00:24:46.817187+0000","last_clean":"2020-03-09T00:24:46.817187+0000","last_became_active":"2020-03-09T00:24:43.052840+0000","last_became_peered":"2020-03-09T00:24:43.052840+0000","last_unstale":"2020-03-09T00:24:46.817187+0000","last_undegraded":"2020-03-09T00:24:46.817187+0000","last_fullsized":"2020-03-09T00:24:46.817187+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1425408,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":1445,"num_write":9,"num_write_kb":1392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2,15,4,5,3],"acting":[13,7,2,15,4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":"16","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.811185+0000","last_change":"2020-03-09T00:24:27.920285+0000","last_active":"2020-03-09T00:24:44.811185+0000","last_peered":"2020-03-09T00:24:44.811185+0000","last_clean":"2020-03-09T00:24:44.811185+0000","last_became_active":"2020-03-09T00:24:27.919842+0000","last_became_peered":"2020-03-09T00:24:27.919842+0000","last_unstale":"2020-03-09T00:24:44.811185+0000","last_undegraded":"2020-03-09T00:24:44.811185+0000","last_fullsized":"2020-03-09T00:24:44.811185+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"35'9","reported_seq":"28","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.960943+0000","last_change":"2020-03-09T00:24:43.219811+0000","last_active":"2020-03-09T00:24:44.960943+0000","last_peered":"2020-03-09T00:24:44.960943+0000","last_clean":"2020-03-09T00:24:44.960943+0000","last_became_active":"2020-03-09T00:24:43.218941+0000","last_became_peered":"2020-03-09T00:24:43.218941+0000","last_unstale":"2020-03-09T00:24:44.960943+0000","last_undegraded":"2020-03-09T00:24:44.960943+0000","last_fullsized":"2020-03-09T00:24:44.960943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2228224,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":9,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,11,15,1,5,0,6],"acting":[12,11,15,1,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"16","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.809336+0000","last_change":"2020-03-09T00:24:27.919033+0000","last_active":"2020-03-09T00:24:44.809336+0000","last_peered":"2020-03-09T00:24:44.809336+0000","last_clean":"2020-03-09T00:24:44.809336+0000","last_became_active":"2020-03-09T00:24:27.918573+0000","last_became_peered":"2020-03-09T00:24:27.918573+0000","last_unstale":"2020-03-09T00:24:44.809336+0000","last_undegraded":"2020-03-09T00:24:44.809336+0000","last_fullsized":"2020-03-09T00:24:44.809336+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'3","reported_seq":"22","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.811082+0000","last_change":"2020-03-09T00:24:22.183631+0000","last_active":"2020-03-09T00:24:44.811082+0000","last_peered":"2020-03-09T00:24:44.811082+0000","last_clean":"2020-03-09T00:24:44.811082+0000","last_became_active":"2020-03-09T00:24:22.183150+0000","last_became_peered":"2020-03-09T00:24:22.183150+0000","last_unstale":"2020-03-09T00:24:44.811082+0000","last_undegraded":"2020-03-09T00:24:44.811082+0000","last_fullsized":"2020-03-09T00:24:44.811082+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:10.644148+0000","last_clean_scrub_stamp":"2020-03-09T00:24:10.644148+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.2","version":"35'5","reported_seq":"27","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.809213+0000","last_change":"2020-03-09T00:24:46.809677+0000","last_active":"2020-03-09T00:24:46.809213+0000","last_peered":"2020-03-09T00:24:46.809213+0000","last_clean":"2020-03-09T00:24:46.809213+0000","last_became_active":"2020-03-09T00:24:43.051704+0000","last_became_peered":"2020-03-09T00:24:43.051704+0000","last_unstale":"2020-03-09T00:24:46.809213+0000","last_undegraded":"2020-03-09T00:24:46.809213+0000","last_fullsized":"2020-03-09T00:24:46.809213+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":278528,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3,4,15,1,11],"acting":[13,12,3,4,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":"16","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.820540+0000","last_change":"2020-03-09T00:24:27.941291+0000","last_active":"2020-03-09T00:24:44.820540+0000","last_peered":"2020-03-09T00:24:44.820540+0000","last_clean":"2020-03-09T00:24:44.820540+0000","last_became_active":"2020-03-09T00:24:27.940904+0000","last_became_peered":"2020-03-09T00:24:27.940904+0000","last_unstale":"2020-03-09T00:24:44.820540+0000","last_undegraded":"2020-03-09T00:24:44.820540+0000","last_fullsized":"2020-03-09T00:24:44.820540+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"35'8","reported_seq":"39","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.818040+0000","last_change":"2020-03-09T00:24:46.816786+0000","last_active":"2020-03-09T00:24:46.818040+0000","last_peered":"2020-03-09T00:24:46.818040+0000","last_clean":"2020-03-09T00:24:46.818040+0000","last_became_active":"2020-03-09T00:24:43.217710+0000","last_became_peered":"2020-03-09T00:24:43.217710+0000","last_unstale":"2020-03-09T00:24:46.818040+0000","last_undegraded":"2020-03-09T00:24:46.818040+0000","last_fullsized":"2020-03-09T00:24:46.818040+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":1158,"num_write":8,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9,5,12,0,13],"acting":[6,11,9,5,12,0,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":"17","reported_epoch":"36","state":"active+clean","last_fresh":"2020-03-09T00:24:45.801216+0000","last_change":"2020-03-09T00:24:27.925113+0000","last_active":"2020-03-09T00:24:45.801216+0000","last_peered":"2020-03-09T00:24:45.801216+0000","last_clean":"2020-03-09T00:24:45.801216+0000","last_became_active":"2020-03-09T00:24:27.924718+0000","last_became_peered":"2020-03-09T00:24:27.924718+0000","last_unstale":"2020-03-09T00:24:45.801216+0000","last_undegraded":"2020-03-09T00:24:45.801216+0000","last_fullsized":"2020-03-09T00:24:45.801216+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"35'2","reported_seq":"19","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.908292+0000","last_change":"2020-03-09T00:24:42.806011+0000","last_active":"2020-03-09T00:24:44.908292+0000","last_peered":"2020-03-09T00:24:44.908292+0000","last_clean":"2020-03-09T00:24:44.908292+0000","last_became_active":"2020-03-09T00:24:42.804848+0000","last_became_peered":"2020-03-09T00:24:42.804848+0000","last_unstale":"2020-03-09T00:24:44.908292+0000","last_undegraded":"2020-03-09T00:24:44.908292+0000","last_fullsized":"2020-03-09T00:24:44.908292+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":589824,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,12,14,1,6,11,13],"acting":[10,12,14,1,6,11,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"35'4","reported_seq":"30","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.818251+0000","last_change":"2020-03-09T00:24:46.817275+0000","last_active":"2020-03-09T00:24:46.818251+0000","last_peered":"2020-03-09T00:24:46.818251+0000","last_clean":"2020-03-09T00:24:46.818251+0000","last_became_active":"2020-03-09T00:24:43.087104+0000","last_became_peered":"2020-03-09T00:24:43.087104+0000","last_unstale":"2020-03-09T00:24:46.818251+0000","last_undegraded":"2020-03-09T00:24:46.818251+0000","last_fullsized":"2020-03-09T00:24:46.818251+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":524288,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":435,"num_write":4,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,9,4,2,5,11],"acting":[14,8,9,4,2,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":"17","reported_epoch":"36","state":"active+clean","last_fresh":"2020-03-09T00:24:45.801937+0000","last_change":"2020-03-09T00:24:27.941155+0000","last_active":"2020-03-09T00:24:45.801937+0000","last_peered":"2020-03-09T00:24:45.801937+0000","last_clean":"2020-03-09T00:24:45.801937+0000","last_became_active":"2020-03-09T00:24:27.940788+0000","last_became_peered":"2020-03-09T00:24:27.940788+0000","last_unstale":"2020-03-09T00:24:45.801937+0000","last_undegraded":"2020-03-09T00:24:45.801937+0000","last_fullsized":"2020-03-09T00:24:45.801937+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"16","reported_epoch":"35","state":"active+clean","last_fresh":"2020-03-09T00:24:44.811611+0000","last_change":"2020-03-09T00:24:27.919488+0000","last_active":"2020-03-09T00:24:44.811611+0000","last_peered":"2020-03-09T00:24:44.811611+0000","last_clean":"2020-03-09T00:24:44.811611+0000","last_became_active":"2020-03-09T00:24:27.919102+0000","last_became_peered":"2020-03-09T00:24:27.919102+0000","last_unstale":"2020-03-09T00:24:44.811611+0000","last_undegraded":"2020-03-09T00:24:44.811611+0000","last_fullsized":"2020-03-09T00:24:44.811611+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'10","reported_seq":"35","reported_epoch":"37","state":"active+clean","last_fresh":"2020-03-09T00:24:46.814954+0000","last_change":"2020-03-09T00:24:46.815721+0000","last_active":"2020-03-09T00:24:46.814954+0000","last_peered":"2020-03-09T00:24:46.814954+0000","last_clean":"2020-03-09T00:24:46.814954+0000","last_became_active":"2020-03-09T00:24:43.086971+0000","last_became_peered":"2020-03-09T00:24:43.086971+0000","last_unstale":"2020-03-09T00:24:46.814954+0000","last_undegraded":"2020-03-09T00:24:46.814954+0000","last_fullsized":"2020-03-09T00:24:46.814954+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1458176,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":10,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,1,15,6,7,9],"acting":[14,13,1,15,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":20955136,"num_objects":48,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":127,"num_read_kb":5914,"num_write":108,"num_write_kb":21456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36671488,"data_stored":36671488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":112,"acting":112,"num_store_stats":16},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11}],"osd_stats":[{"osd":15,"up_from":24,"seq":103079215112,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051644,"kb_used_data":3060,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320196,"statfs":{"total":96636764160,"available":95559880704,"internally_reserved":1073741824,"allocated":3133440,"data_stored":2963778,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,4,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.325}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3580000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.47}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]}]},{"osd":14,"up_from":24,"seq":103079215111,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052124,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319716,"statfs":{"total":96636764160,"available":95559389184,"internally_reserved":1073741824,"allocated":3624960,"data_stored":3453186,"data_compressed":24488,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2570000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9830000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7230000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8120000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2970000000000002}]}]},{"osd":13,"up_from":24,"seq":103079215112,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050824,"kb_used_data":2240,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321016,"statfs":{"total":96636764160,"available":95560720384,"internally_reserved":1073741824,"allocated":2293760,"data_stored":2124098,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7210000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.972}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.849}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7770000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.302}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0570000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.734}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7959999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4119999999999999}]}]},{"osd":12,"up_from":24,"seq":103079215111,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051532,"kb_used_data":2948,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320308,"statfs":{"total":96636764160,"available":95559995392,"internally_reserved":1073741824,"allocated":3018752,"data_stored":2849090,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,4,5,6,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5980000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8480000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9530000000000001}]}]},{"osd":11,"up_from":24,"seq":103079215112,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051968,"kb_used_data":3384,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319872,"statfs":{"total":96636764160,"available":95559548928,"internally_reserved":1073741824,"allocated":3465216,"data_stored":3295554,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2909999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0590000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4340000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]}]},{"osd":9,"up_from":24,"seq":103079215112,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050668,"kb_used_data":2084,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321172,"statfs":{"total":96636764160,"available":95560880128,"internally_reserved":1073741824,"allocated":2134016,"data_stored":1964354,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4390000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.615}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.657}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]}]},{"osd":8,"up_from":24,"seq":103079215111,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050264,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321576,"statfs":{"total":96636764160,"available":95561293824,"internally_reserved":1073741824,"allocated":1720320,"data_stored":1548546,"data_compressed":24488,"data_compressed_allocated":53248,"data_compressed_original":106496,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,7,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]}]},{"osd":7,"up_from":24,"seq":103079215112,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050228,"kb_used_data":1644,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321612,"statfs":{"total":96636764160,"available":95561330688,"internally_reserved":1073741824,"allocated":1683456,"data_stored":1513794,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.659}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5680000000000001}]}]},{"osd":6,"up_from":24,"seq":103079215112,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051784,"kb_used_data":3200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320056,"statfs":{"total":96636764160,"available":95559737344,"internally_reserved":1073741824,"allocated":3276800,"data_stored":3107138,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]}]},{"osd":5,"up_from":24,"seq":103079215113,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050692,"kb_used_data":2108,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321148,"statfs":{"total":96636764160,"available":95560855552,"internally_reserved":1073741824,"allocated":2158592,"data_stored":1988930,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3460000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3050000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.619}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.825}]}]},{"osd":0,"up_from":24,"seq":103079215111,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050816,"kb_used_data":2232,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321024,"statfs":{"total":96636764160,"available":95560728576,"internally_reserved":1073741824,"allocated":2285568,"data_stored":2115906,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1539999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]}]},{"osd":10,"up_from":23,"seq":98784247816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050528,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321312,"statfs":{"total":96636764160,"available":95561023488,"internally_reserved":1073741824,"allocated":1990656,"data_stored":1820994,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,6,7,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4550000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9710000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]}]},{"osd":3,"up_from":24,"seq":103079215112,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050056,"kb_used_data":1472,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321784,"statfs":{"total":96636764160,"available":95561506816,"internally_reserved":1073741824,"allocated":1507328,"data_stored":1337666,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7130000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]}]},{"osd":1,"up_from":24,"seq":103079215111,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052124,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93319716,"statfs":{"total":96636764160,"available":95559389184,"internally_reserved":1073741824,"allocated":3624960,"data_stored":3455298,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.841}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9039999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.101}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0350000000000001}]}]},{"osd":2,"up_from":24,"seq":103079215111,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050988,"kb_used_data":2404,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320852,"statfs":{"total":96636764160,"available":95560552448,"internally_reserved":1073741824,"allocated":2461696,"data_stored":2292034,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0129999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]}]},{"osd":4,"up_from":24,"seq":103079215112,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1051544,"kb_used_data":2960,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93320296,"statfs":{"total":96636764160,"available":95559983104,"internally_reserved":1073741824,"allocated":3031040,"data_stored":2861378,"data_compressed":22539,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]}]}]}} 2020-03-09T00:24:49.168 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2020-03-09T00:24:49.169 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:49.184 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:49.205 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:49.226 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:49.235 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:49.233+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:49.247 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:49.261 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:49.261 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:49.267 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:49.270 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:24:49.289 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:49.309 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:49.311 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:49.309+0000 7fc9cb59e700 0 -- 172.21.15.148:0/3280023356 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7fc9cc18b310 msgr2=0x7fc9cc18d700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:49.330 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:49.331 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.331 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.331 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.331 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.331 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.332 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.333 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.333 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.333 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.333 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:24:49.336 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:49.335+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:49.396 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:49.396 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:49.404 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:24:49.437 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:49.436+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:49.520 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:49.520 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:49.528 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:24:49.571 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:49.569+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:49.638 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:49.637+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:49.643 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:49.643 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:49.651 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:24:49.739 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:49.738+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:49.778 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:49.778 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:49.787 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:24:49.828 INFO:tasks.rados.rados.0.smithi104.stdout:77: done (1 left) 2020-03-09T00:24:49.828 INFO:tasks.rados.rados.0.smithi104.stdout:78: done (0 left) 2020-03-09T00:24:49.828 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 8 current snap is 4 2020-03-09T00:24:49.828 INFO:tasks.rados.rados.0.smithi104.stdout:79: seq_num 56 ranges {1048576=573440,1622016=376832} 2020-03-09T00:24:49.836 INFO:tasks.rados.rados.0.smithi104.stdout:79: writing smithi10428818-8 from 1048576 to 1622016 tid 1 2020-03-09T00:24:49.839 INFO:tasks.rados.rados.0.smithi104.stdout:79: writing smithi10428818-8 from 1622016 to 1998848 tid 2 2020-03-09T00:24:49.839 INFO:tasks.rados.rados.0.smithi104.stdout:80: snap_create 2020-03-09T00:24:49.841 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:49.840+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:49.849 INFO:tasks.rados.rados.0.smithi104.stdout:79: finishing write tid 1 to smithi10428818-8 2020-03-09T00:24:49.849 INFO:tasks.rados.rados.0.smithi104.stdout:79: finishing write tid 2 to smithi10428818-8 2020-03-09T00:24:49.851 INFO:tasks.rados.rados.0.smithi104.stdout:79: finishing write tid 4 to smithi10428818-8 2020-03-09T00:24:49.853 INFO:tasks.rados.rados.0.smithi104.stdout:79: finishing write tid 5 to smithi10428818-8 2020-03-09T00:24:49.853 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 8 v 12 (ObjNum 56 snap 4 seq_num 56) dirty exists 2020-03-09T00:24:49.853 INFO:tasks.rados.rados.0.smithi104.stdout:79: left oid 8 (ObjNum 56 snap 4 seq_num 56) 2020-03-09T00:24:49.942 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:49.941+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:49.944 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:49.944 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:49.953 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:24:50.043 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:50.042+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:50.067 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:50.068 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:50.076 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:24:50.144 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:50.142+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:50.216 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:50.216 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:50.225 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:24:50.245 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:50.243+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:50.346 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:50.345+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:50.348 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:50.349 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:50.357 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:24:50.447 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:50.445+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:24:50.481 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:50.481 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:50.490 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:24:50.549 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:50.546+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:50.618 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:50.618 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:50.626 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:24:50.648 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:50.646+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:50.749 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:50.748+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:50.782 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:50.782 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:50.791 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:24:50.828 INFO:tasks.rados.rados.0.smithi104.stdout:79: done (1 left) 2020-03-09T00:24:50.829 INFO:tasks.rados.rados.0.smithi104.stdout:80: done (0 left) 2020-03-09T00:24:50.829 INFO:tasks.rados.rados.0.smithi104.stdout:81: read oid 37 snap 4 2020-03-09T00:24:50.829 INFO:tasks.rados.rados.0.smithi104.stdout:81: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:24:50.829 INFO:tasks.rados.rados.0.smithi104.stdout:82: copy_from oid 17 from oid 32 current snap is 5 2020-03-09T00:24:50.829 INFO:tasks.rados.rados.0.smithi104.stdout:83: snap_remove snap 3 2020-03-09T00:24:50.835 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-03-09T00:24:50.850 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:50.849+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:50.916 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:50.916 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:50.925 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:24:50.951 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:50.950+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:51.049 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:51.050 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:51.052 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:51.049+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:51.059 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:24:51.153 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:51.152+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:51.214 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:51.223 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:24:51.254 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:51.253+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:51.337 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:51.337 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:51.341 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:24:51.345 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-03-09T00:24:51.389 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:51.388+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:51.456 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:51.455+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:24:51.472 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:51.473 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:51.473 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:51.482 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-03-09T00:24:51.494 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:51.514 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:51.534 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:51.553 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:51.558 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:51.556+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:51.574 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:51.593 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:51.613 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:51.632 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:51.636 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:51.636 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:51.636 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:24:51.636 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:24:51.636 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:51.644 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-03-09T00:24:51.653 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:51.658 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:51.657+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:51.673 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:51.693 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:51.713 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:51.733 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:51.753 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:51.759 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:51.758+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:24:51.762 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:51.762 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:24:51.762 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:24:51.762 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:24:51.762 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.762 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:174 3.1s0 3:8a6d1775:::smithi10428818-12:head [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:24:51.762 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.810668+0000", 2020-03-09T00:24:51.763 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9489304970000001, 2020-03-09T00:24:51.763 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068006910000000002, 2020-03-09T00:24:51.763 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.763 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:51.763 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:51.763 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:51.763 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:51.764 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 174 2020-03-09T00:24:51.764 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.764 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.764 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.764 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.764 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810668+0000", 2020-03-09T00:24:51.764 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810668+0000", 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2623999999999997e-05 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.765 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.766 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810710+0000", 2020-03-09T00:24:51.766 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7649999999999996e-06 2020-03-09T00:24:51.766 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.766 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.766 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.766 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810717+0000", 2020-03-09T00:24:51.766 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2300000000000001e-05 2020-03-09T00:24:51.767 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.767 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.767 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.767 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810729+0000", 2020-03-09T00:24:51.767 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0500000000000003e-06 2020-03-09T00:24:51.767 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.768 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.768 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810736+0000", 2020-03-09T00:24:51.768 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.5187999999999999e-05 2020-03-09T00:24:51.768 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.768 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810811+0000", 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8220000000000001e-06 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810818+0000", 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0032096389999999998 2020-03-09T00:24:51.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.770 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.770 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.770 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814028+0000", 2020-03-09T00:24:51.770 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010378900000000001 2020-03-09T00:24:51.770 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.770 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.770 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:51.771 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814132+0000", 2020-03-09T00:24:51.771 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033365140000000001 2020-03-09T00:24:51.771 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.771 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.771 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.771 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817468+0000", 2020-03-09T00:24:51.772 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068006910000000002 2020-03-09T00:24:51.772 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.772 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.772 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.772 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.772 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.772 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:175 3.1s0 3:8a6d1775:::smithi10428818-12:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:24:51.773 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.810777+0000", 2020-03-09T00:24:51.773 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9488208480000004, 2020-03-09T00:24:51.773 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007306959, 2020-03-09T00:24:51.773 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.773 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:51.773 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:51.774 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:51.774 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:51.774 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 175 2020-03-09T00:24:51.774 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.774 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.774 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.774 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810777+0000", 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810777+0000", 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0694e-05 2020-03-09T00:24:51.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.776 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.776 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.776 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810818+0000", 2020-03-09T00:24:51.776 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.7500000000000001e-06 2020-03-09T00:24:51.776 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.776 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.776 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.777 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810822+0000", 2020-03-09T00:24:51.777 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7689999999999999e-06 2020-03-09T00:24:51.777 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.777 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.777 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.777 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810828+0000", 2020-03-09T00:24:51.777 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9900000000000002e-06 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810831+0000", 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1592000000000003e-05 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.778 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.779 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810893+0000", 2020-03-09T00:24:51.779 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2199999999999998e-06 2020-03-09T00:24:51.779 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.779 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.779 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:51.779 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810902+0000", 2020-03-09T00:24:51.779 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003716488 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814619+0000", 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000108533 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.780 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:51.781 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814727+0000", 2020-03-09T00:24:51.781 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033569229999999999 2020-03-09T00:24:51.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.781 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.781 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.781 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818084+0000", 2020-03-09T00:24:51.781 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007306959 2020-03-09T00:24:51.782 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.782 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.782 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.782 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.782 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.782 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:176 3.1s0 3:8a6d1775:::smithi10428818-12:head [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:24:51.782 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.810868+0000", 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9487296140000003, 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007708576, 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:51.783 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 176 2020-03-09T00:24:51.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.784 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.784 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.784 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.784 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810868+0000", 2020-03-09T00:24:51.784 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.785 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.785 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810868+0000", 2020-03-09T00:24:51.785 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1648000000000002e-05 2020-03-09T00:24:51.785 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.785 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810940+0000", 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2149999999999998e-06 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810946+0000", 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5269999999999998e-06 2020-03-09T00:24:51.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810955+0000", 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0589999999999996e-06 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.787 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.810959+0000", 2020-03-09T00:24:51.788 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8393000000000002e-05 2020-03-09T00:24:51.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.788 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.788 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811007+0000", 2020-03-09T00:24:51.788 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3229999999999998e-06 2020-03-09T00:24:51.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811016+0000", 2020-03-09T00:24:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041240310000000002 2020-03-09T00:24:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.789 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.790 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815140+0000", 2020-03-09T00:24:51.790 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011424700000000001 2020-03-09T00:24:51.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.791 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:51.791 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815254+0000", 2020-03-09T00:24:51.791 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033231329999999998 2020-03-09T00:24:51.791 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.792 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.792 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.792 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818577+0000", 2020-03-09T00:24:51.793 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007708576 2020-03-09T00:24:51.793 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.793 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.793 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.793 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.793 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.794 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s0 37/32 ECSubRead(tid=21, to_read={3:8a6d1775:::smithi10428818-12:head=0,49152,0}, subchunks={3:8a6d1775:::smithi10428818-12:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:51.794 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.814426+0000", 2020-03-09T00:24:51.794 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9451717930000001, 2020-03-09T00:24:51.794 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001404602, 2020-03-09T00:24:51.794 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814426+0000", 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.795 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814426+0000", 2020-03-09T00:24:51.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814426+0000", 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814426+0000", 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6969999999999999e-06 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.798 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.798 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814435+0000", 2020-03-09T00:24:51.798 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0419999999999997e-06 2020-03-09T00:24:51.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.798 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.799 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814442+0000", 2020-03-09T00:24:51.799 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012401790000000001 2020-03-09T00:24:51.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.799 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815682+0000", 2020-03-09T00:24:51.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000148684 2020-03-09T00:24:51.800 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.800 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.800 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.800 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815831+0000", 2020-03-09T00:24:51.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001404602 2020-03-09T00:24:51.801 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.801 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.801 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.801 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s0 37/32 ECSubRead(tid=22, to_read={3:8a6d1775:::smithi10428818-12:head=0,49152,0}, subchunks={3:8a6d1775:::smithi10428818-12:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:51.801 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.814992+0000", 2020-03-09T00:24:51.802 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9446057520000002, 2020-03-09T00:24:51.802 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099894999999999997, 2020-03-09T00:24:51.802 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.802 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.802 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.802 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.802 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.803 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814992+0000", 2020-03-09T00:24:51.803 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.803 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.803 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.803 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814992+0000", 2020-03-09T00:24:51.803 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.804 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.804 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814992+0000", 2020-03-09T00:24:51.804 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.814992+0000", 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8037999999999999e-05 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815010+0000", 2020-03-09T00:24:51.805 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9960000000000005e-06 2020-03-09T00:24:51.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.806 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.806 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815020+0000", 2020-03-09T00:24:51.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00084799699999999997 2020-03-09T00:24:51.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815868+0000", 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012291900000000001 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815991+0000", 2020-03-09T00:24:51.807 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099894999999999997 2020-03-09T00:24:51.808 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.808 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.808 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.808 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.808 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.1s0 37/32 ECSubReadReply(tid=21, attrs_read=0))", 2020-03-09T00:24:51.808 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:51.809 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9437748690000003, 2020-03-09T00:24:51.809 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016600429999999999, 2020-03-09T00:24:51.809 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.809 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.809 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.809 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.809 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.811 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.811 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:51.811 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.811 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.812 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815823+0000", 2020-03-09T00:24:51.812 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011059729999999999 2020-03-09T00:24:51.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.812 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.812 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816929+0000", 2020-03-09T00:24:51.812 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9409999999999998e-06 2020-03-09T00:24:51.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.813 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.813 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.813 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816934+0000", 2020-03-09T00:24:51.813 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000193289 2020-03-09T00:24:51.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.813 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.814 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.814 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817127+0000", 2020-03-09T00:24:51.814 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00035584000000000001 2020-03-09T00:24:51.814 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.814 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.814 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.814 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817483+0000", 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016600429999999999 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.1s0 37/32 ECSubReadReply(tid=22, attrs_read=0))", 2020-03-09T00:24:51.815 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.815987+0000", 2020-03-09T00:24:51.816 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9436112120000004, 2020-03-09T00:24:51.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001572972, 2020-03-09T00:24:51.816 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.816 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.816 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.816 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.816 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815987+0000", 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815987+0000", 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815987+0000", 2020-03-09T00:24:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815987+0000", 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00095788100000000005 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816945+0000", 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0139999999999999e-06 2020-03-09T00:24:51.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816948+0000", 2020-03-09T00:24:51.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000562503 2020-03-09T00:24:51.820 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.821 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817510+0000", 2020-03-09T00:24:51.821 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9573999999999997e-05 2020-03-09T00:24:51.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.821 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817560+0000", 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001572972 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.822 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.1s0 37/32 ECSubReadReply(tid=23, attrs_read=0))", 2020-03-09T00:24:51.823 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.816142+0000", 2020-03-09T00:24:51.823 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9434563770000004, 2020-03-09T00:24:51.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001482026, 2020-03-09T00:24:51.823 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.823 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.823 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.824 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.824 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816142+0000", 2020-03-09T00:24:51.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.824 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816142+0000", 2020-03-09T00:24:51.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816142+0000", 2020-03-09T00:24:51.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.826 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.827 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816142+0000", 2020-03-09T00:24:51.827 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00081622400000000005 2020-03-09T00:24:51.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.827 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816958+0000", 2020-03-09T00:24:51.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1e-06 2020-03-09T00:24:51.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816961+0000", 2020-03-09T00:24:51.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00062246000000000003 2020-03-09T00:24:51.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817583+0000", 2020-03-09T00:24:51.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0241999999999999e-05 2020-03-09T00:24:51.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.830 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.830 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817624+0000", 2020-03-09T00:24:51.830 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001482026 2020-03-09T00:24:51.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.830 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.830 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.831 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.1s0 35/32 ECSubReadReply(tid=22, attrs_read=0))", 2020-03-09T00:24:51.831 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.816158+0000", 2020-03-09T00:24:51.831 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9434402610000001, 2020-03-09T00:24:51.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00088236200000000003, 2020-03-09T00:24:51.831 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.831 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816158+0000", 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.833 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816158+0000", 2020-03-09T00:24:51.833 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7819999999999998e-05 2020-03-09T00:24:51.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816226+0000", 2020-03-09T00:24:51.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8689999999999998e-06 2020-03-09T00:24:51.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816232+0000", 2020-03-09T00:24:51.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0999999999999998e-06 2020-03-09T00:24:51.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816240+0000", 2020-03-09T00:24:51.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.224e-06 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816243+0000", 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00075075399999999996 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.816994+0000", 2020-03-09T00:24:51.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6594999999999997e-05 2020-03-09T00:24:51.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817040+0000", 2020-03-09T00:24:51.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00088236200000000003 2020-03-09T00:24:51.838 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.838 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.838 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.839 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.1s0 35/32 ECSubReadReply(tid=22, attrs_read=0))", 2020-03-09T00:24:51.839 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.817302+0000", 2020-03-09T00:24:51.839 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9422957639999998, 2020-03-09T00:24:51.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079413699999999999, 2020-03-09T00:24:51.840 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.840 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.840 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817302+0000", 2020-03-09T00:24:51.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817302+0000", 2020-03-09T00:24:51.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010428 2020-03-09T00:24:51.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817407+0000", 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0411000000000001e-05 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817417+0000", 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0037000000000001e-05 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817437+0000", 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.7110000000000001e-06 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817441+0000", 2020-03-09T00:24:51.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00033282100000000002 2020-03-09T00:24:51.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817773+0000", 2020-03-09T00:24:51.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00032287700000000001 2020-03-09T00:24:51.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818096+0000", 2020-03-09T00:24:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00079413699999999999 2020-03-09T00:24:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpReadReply(3.1s0 37/32 ECSubReadReply(tid=23, attrs_read=0))", 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.817614+0000", 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.9419835760000002, 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000973681, 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.847 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817614+0000", 2020-03-09T00:24:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817614+0000", 2020-03-09T00:24:51.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010006899999999999 2020-03-09T00:24:51.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817714+0000", 2020-03-09T00:24:51.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.876e-06 2020-03-09T00:24:51.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817724+0000", 2020-03-09T00:24:51.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2133999999999999e-05 2020-03-09T00:24:51.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817737+0000", 2020-03-09T00:24:51.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7339999999999999e-06 2020-03-09T00:24:51.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.817741+0000", 2020-03-09T00:24:51.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00047638699999999999 2020-03-09T00:24:51.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818218+0000", 2020-03-09T00:24:51.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00037048099999999998 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818588+0000", 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000973681 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.853 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.854 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(osd.10.23:1 3.1s0 3:8a6d1775:::smithi10428818-12:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:51.854 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.814386+0000", 2020-03-09T00:24:51.854 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.9452122219999999, 2020-03-09T00:24:51.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067838029999999997, 2020-03-09T00:24:51.854 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.855 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:51.855 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814386+0000", 2020-03-09T00:24:51.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.856 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.856 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814386+0000", 2020-03-09T00:24:51.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3907999999999998e-05 2020-03-09T00:24:51.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814460+0000", 2020-03-09T00:24:51.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6405999999999998e-05 2020-03-09T00:24:51.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814476+0000", 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3822000000000001e-05 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814490+0000", 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7399999999999998e-06 2020-03-09T00:24:51.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814497+0000", 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7738999999999997e-05 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814554+0000", 2020-03-09T00:24:51.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1499999999999999e-06 2020-03-09T00:24:51.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2020-03-09T00:24:51.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.814563+0000", 2020-03-09T00:24:51.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017988819999999999 2020-03-09T00:24:51.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816361+0000", 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013230800000000001 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:51.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816494+0000", 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046758479999999998 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821170+0000", 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067838029999999997 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.862 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.863 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.863 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s6 38/32 ECSubWrite(tid=46, reqid=client.4751.0:195, at_version=38'16, trim_to=0'0, roll_forward_to=35'14))", 2020-03-09T00:24:51.863 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.815409+0000", 2020-03-09T00:24:51.863 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.9441891309999999, 2020-03-09T00:24:51.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099192799999999999, 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815409+0000", 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815409+0000", 2020-03-09T00:24:51.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3324999999999997e-05 2020-03-09T00:24:51.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815442+0000", 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0568e-05 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815463+0000", 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4429999999999992e-06 2020-03-09T00:24:51.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815472+0000", 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3369e-05 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815486+0000", 2020-03-09T00:24:51.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7021999999999997e-05 2020-03-09T00:24:51.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815543+0000", 2020-03-09T00:24:51.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2694999999999999e-05 2020-03-09T00:24:51.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:51.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.815565+0000", 2020-03-09T00:24:51.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00078494299999999997 2020-03-09T00:24:51.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.870 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:51.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816350+0000", 2020-03-09T00:24:51.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0562999999999997e-05 2020-03-09T00:24:51.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816401+0000", 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00099192799999999999 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpRead(3.1s0 38/32 ECSubRead(tid=24, to_read={3:8a6d1775:::smithi10428818-12:head=0,49152,0}, subchunks={3:8a6d1775:::smithi10428818-12:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.816920+0000", 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.942677953, 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00082168200000000001, 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.872 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816920+0000", 2020-03-09T00:24:51.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816920+0000", 2020-03-09T00:24:51.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816920+0000", 2020-03-09T00:24:51.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816920+0000", 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1956e-05 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816932+0000", 2020-03-09T00:24:51.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1159999999999999e-06 2020-03-09T00:24:51.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816940+0000", 2020-03-09T00:24:51.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00063836299999999997 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817578+0000", 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016324699999999999 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817742+0000", 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00082168200000000001 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.878 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/32 ECSubWrite(tid=20, reqid=client.4751.0:190, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:24:51.879 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.823052+0000", 2020-03-09T00:24:51.879 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.9365464929999998, 2020-03-09T00:24:51.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026814270000000001, 2020-03-09T00:24:51.879 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.879 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.879 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823052+0000", 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823052+0000", 2020-03-09T00:24:51.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4794000000000003e-05 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823126+0000", 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5343000000000001e-05 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823152+0000", 2020-03-09T00:24:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.63e-05 2020-03-09T00:24:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823168+0000", 2020-03-09T00:24:51.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5699999999999998e-06 2020-03-09T00:24:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823175+0000", 2020-03-09T00:24:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015091400000000001 2020-03-09T00:24:51.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823325+0000", 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4521999999999999e-05 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:51.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823340+0000", 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023288530000000001 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.825669+0000", 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4131e-05 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.825733+0000", 2020-03-09T00:24:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026814270000000001 2020-03-09T00:24:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.886 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.4s4 38/32 ECSubWrite(tid=21, reqid=client.4751.0:190, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.826922+0000", 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.9326759930000001, 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00083418999999999995, 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.887 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826922+0000", 2020-03-09T00:24:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826922+0000", 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1421999999999995e-05 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826993+0000", 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3978000000000001e-05 2020-03-09T00:24:51.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827007+0000", 2020-03-09T00:24:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6020000000000002e-05 2020-03-09T00:24:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827033+0000", 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9449999999999993e-06 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827041+0000", 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7360000000000001e-05 2020-03-09T00:24:51.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827089+0000", 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1823e-05 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:51.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827101+0000", 2020-03-09T00:24:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00060462899999999997 2020-03-09T00:24:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827705+0000", 2020-03-09T00:24:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1013000000000001e-05 2020-03-09T00:24:51.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827756+0000", 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00083418999999999995 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.895 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.5s1 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:51.895 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.827438+0000", 2020-03-09T00:24:51.895 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.932160192, 2020-03-09T00:24:51.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0031241929999999999, 2020-03-09T00:24:51.895 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.895 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827438+0000", 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827438+0000", 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023304599999999999 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827671+0000", 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4394999999999999e-05 2020-03-09T00:24:51.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827695+0000", 2020-03-09T00:24:51.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6382999999999999e-05 2020-03-09T00:24:51.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827712+0000", 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6690000000000006e-06 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827719+0000", 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016007599999999999 2020-03-09T00:24:51.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827879+0000", 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2442e-05 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:51.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827892+0000", 2020-03-09T00:24:51.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002622274 2020-03-09T00:24:51.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:51.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.830514+0000", 2020-03-09T00:24:51.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7908000000000003e-05 2020-03-09T00:24:51.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.830562+0000", 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0031241929999999999 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.5s1 38/32 ECSubWrite(tid=11, reqid=client.4751.0:197, at_version=38'7, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.832806+0000", 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.9267918970000002, 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020194129999999999, 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.903 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832806+0000", 2020-03-09T00:24:51.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832806+0000", 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2144999999999994e-05 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832888+0000", 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4369000000000001e-05 2020-03-09T00:24:51.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832903+0000", 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1219999999999999e-05 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832914+0000", 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3320000000000002e-06 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832918+0000", 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0454000000000003e-05 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832959+0000", 2020-03-09T00:24:51.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1112e-05 2020-03-09T00:24:51.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:51.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.832970+0000", 2020-03-09T00:24:51.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001807052 2020-03-09T00:24:51.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:51.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.834777+0000", 2020-03-09T00:24:51.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8729000000000001e-05 2020-03-09T00:24:51.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.834826+0000", 2020-03-09T00:24:51.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020194129999999999 2020-03-09T00:24:51.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.911 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.911 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.5s1 38/32 ECSubWrite(tid=13, reqid=client.4751.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=38'8))", 2020-03-09T00:24:51.912 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.841160+0000", 2020-03-09T00:24:51.912 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.918438417, 2020-03-09T00:24:51.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00091343900000000005, 2020-03-09T00:24:51.912 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.912 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.912 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841160+0000", 2020-03-09T00:24:51.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841160+0000", 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0728000000000005e-05 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841230+0000", 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5988e-05 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841246+0000", 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9647999999999999e-05 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841266+0000", 2020-03-09T00:24:51.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1509999999999993e-06 2020-03-09T00:24:51.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841275+0000", 2020-03-09T00:24:51.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9463999999999999e-05 2020-03-09T00:24:51.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841325+0000", 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6663e-05 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841341+0000", 2020-03-09T00:24:51.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00067087400000000001 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.842012+0000", 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0922999999999999e-05 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.842073+0000", 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00091343900000000005 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.920 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.bs1 39/32 ECSubWrite(tid=22, reqid=client.4751.0:210, at_version=39'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:51.921 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:48.828417+0000", 2020-03-09T00:24:51.921 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.931180592, 2020-03-09T00:24:51.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0043263549999999996, 2020-03-09T00:24:51.921 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:51.921 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:51.921 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:51.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828417+0000", 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828417+0000", 2020-03-09T00:24:51.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015712899999999999 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828575+0000", 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9244e-05 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:51.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828604+0000", 2020-03-09T00:24:51.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7915e-05 2020-03-09T00:24:51.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:51.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828622+0000", 2020-03-09T00:24:51.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0530000000000001e-05 2020-03-09T00:24:51.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:51.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828632+0000", 2020-03-09T00:24:51.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9536999999999993e-05 2020-03-09T00:24:51.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828722+0000", 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4335000000000001e-05 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828736+0000", 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039523869999999999 2020-03-09T00:24:51.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.832688+0000", 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5278000000000002e-05 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:51.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.832744+0000", 2020-03-09T00:24:51.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0043263549999999996 2020-03-09T00:24:51.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.929 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.929 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:51.929 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:51.929 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:51.933 INFO:tasks.rados.rados.0.smithi104.stdout:83: done (2 left) 2020-03-09T00:24:51.933 INFO:tasks.rados.rados.0.smithi104.stdout:84: delete oid 9 current snap is 5 2020-03-09T00:24:51.933 INFO:tasks.rados.rados.0.smithi104.stdout:81: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:24:51.933 INFO:tasks.rados.rados.0.smithi104.stdout:82: finishing copy_from to smithi10428818-17 2020-03-09T00:24:51.933 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 17 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-03-09T00:24:51.934 INFO:tasks.rados.rados.0.smithi104.stdout:82: finishing copy_from racing read to smithi10428818-17 2020-03-09T00:24:51.934 INFO:tasks.rados.rados.0.smithi104.stdout:81: done (2 left) 2020-03-09T00:24:51.934 INFO:tasks.rados.rados.0.smithi104.stdout:82: done (1 left) 2020-03-09T00:24:51.934 INFO:tasks.rados.rados.0.smithi104.stdout:84: done (0 left) 2020-03-09T00:24:51.934 INFO:tasks.rados.rados.0.smithi104.stdout:85: read oid 49 snap -1 2020-03-09T00:24:51.934 INFO:tasks.rados.rados.0.smithi104.stdout:85: expect (ObjNum 48 snap 0 seq_num 48) 2020-03-09T00:24:51.934 INFO:tasks.rados.rados.0.smithi104.stdout:86: setattr oid 7 current snap is 5 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:87: read oid 36 snap 4 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:87: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:88: read oid 10 snap -1 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:88: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:89: snap_create 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 7 v 17 (ObjNum -327083472 snap 22099 seq_num 0) dirty exists 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:87: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:24:51.935 INFO:tasks.rados.rados.0.smithi104.stdout:88: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:24:51.936 INFO:tasks.rados.rados.0.smithi104.stdout:85: expect (ObjNum 48 snap 0 seq_num 48) 2020-03-09T00:24:51.936 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:51.772+0000 7f9f56d9d700 0 -- 172.21.15.148:0/1930292026 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7f9f5818b130 msgr2=0x7f9f5818d520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:51.936 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.936 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.936 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.936 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.936 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.937 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.938 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.938 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:24:51.938 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:51.938 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is set 2020-03-09T00:24:51.938 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:51.929+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:51.939 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:24:52.032 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:52.030+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:52.075 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:52.075 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:52.075 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:24:52.075 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:52.084 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:24:52.133 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:52.131+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:52.208 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:52.208 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:52.209 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:24:52.209 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:24:52.209 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:52.217 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:24:52.233 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:52.231+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:52.334 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:52.333+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:52.342 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:24:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:24:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:24:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.1s2 37/32 ECSubRead(tid=21, to_read={3:8a6d1775:::smithi10428818-12:head=0,49152,0}, subchunks={3:8a6d1775:::smithi10428818-12:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.814601+0000", 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5249771809999997, 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042941099999999999, 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.814601+0000", 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.814601+0000", 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.91e-05 2020-03-09T00:24:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.814660+0000", 2020-03-09T00:24:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1909000000000001e-05 2020-03-09T00:24:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.814672+0000", 2020-03-09T00:24:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9088000000000001e-05 2020-03-09T00:24:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.814691+0000", 2020-03-09T00:24:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8249999999999999e-06 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.814698+0000", 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3067999999999999e-05 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.814771+0000", 2020-03-09T00:24:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025942100000000002 2020-03-09T00:24:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815030+0000", 2020-03-09T00:24:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042941099999999999 2020-03-09T00:24:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.7s2 37/32 ECSubRead(tid=29, to_read={3:e7e20030:::smithi10428818-9:head=0,36864,0}, subchunks={3:e7e20030:::smithi10428818-9:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.815142+0000", 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5244359530000002, 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060405300000000001, 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815142+0000", 2020-03-09T00:24:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815142+0000", 2020-03-09T00:24:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9238000000000004e-05 2020-03-09T00:24:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815212+0000", 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3207000000000001e-05 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815225+0000", 2020-03-09T00:24:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9114e-05 2020-03-09T00:24:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815244+0000", 2020-03-09T00:24:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.043e-05 2020-03-09T00:24:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815254+0000", 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1235999999999994e-05 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815315+0000", 2020-03-09T00:24:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043082800000000002 2020-03-09T00:24:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815746+0000", 2020-03-09T00:24:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060405300000000001 2020-03-09T00:24:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.7s2 37/32 ECSubRead(tid=30, to_read={3:e7e20030:::smithi10428818-9:head=0,36864,0}, subchunks={3:e7e20030:::smithi10428818-9:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.815317+0000", 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5242612250000001, 2020-03-09T00:24:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000734182, 2020-03-09T00:24:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815317+0000", 2020-03-09T00:24:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815317+0000", 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1169999999999999e-05 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815378+0000", 2020-03-09T00:24:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9670000000000007e-06 2020-03-09T00:24:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815387+0000", 2020-03-09T00:24:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6841999999999999e-05 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815404+0000", 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4359999999999999e-06 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815410+0000", 2020-03-09T00:24:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036839099999999998 2020-03-09T00:24:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815779+0000", 2020-03-09T00:24:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027237600000000001 2020-03-09T00:24:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816051+0000", 2020-03-09T00:24:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000734182 2020-03-09T00:24:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.1s2 37/32 ECSubRead(tid=22, to_read={3:8a6d1775:::smithi10428818-12:head=0,49152,0}, subchunks={3:8a6d1775:::smithi10428818-12:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.815958+0000", 2020-03-09T00:24:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5236205289999996, 2020-03-09T00:24:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080555100000000005, 2020-03-09T00:24:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815958+0000", 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815958+0000", 2020-03-09T00:24:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9735999999999999e-05 2020-03-09T00:24:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816017+0000", 2020-03-09T00:24:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1460999999999999e-05 2020-03-09T00:24:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816029+0000", 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7750000000000001e-05 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816047+0000", 2020-03-09T00:24:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1740000000000003e-06 2020-03-09T00:24:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816054+0000", 2020-03-09T00:24:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054290299999999996 2020-03-09T00:24:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816597+0000", 2020-03-09T00:24:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000166527 2020-03-09T00:24:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816763+0000", 2020-03-09T00:24:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080555100000000005 2020-03-09T00:24:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.1s2 37/32 ECSubRead(tid=23, to_read={3:8a6d1775:::smithi10428818-12:head=0,49152,0}, subchunks={3:8a6d1775:::smithi10428818-12:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.816122+0000", 2020-03-09T00:24:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5234565279999996, 2020-03-09T00:24:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087155499999999999, 2020-03-09T00:24:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816122+0000", 2020-03-09T00:24:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816122+0000", 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6910000000000002e-05 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816179+0000", 2020-03-09T00:24:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0321e-05 2020-03-09T00:24:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816189+0000", 2020-03-09T00:24:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5569999999999998e-05 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816204+0000", 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0820000000000004e-06 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816211+0000", 2020-03-09T00:24:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058654099999999999 2020-03-09T00:24:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816797+0000", 2020-03-09T00:24:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000196131 2020-03-09T00:24:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816993+0000", 2020-03-09T00:24:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087155499999999999 2020-03-09T00:24:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.7s2 37/32 ECSubRead(tid=31, to_read={3:e7e20030:::smithi10428818-9:head=0,36864,0}, subchunks={3:e7e20030:::smithi10428818-9:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.816269+0000", 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5233096819999998, 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056926199999999996, 2020-03-09T00:24:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.380 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.380 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816269+0000", 2020-03-09T00:24:52.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816269+0000", 2020-03-09T00:24:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6692000000000003e-05 2020-03-09T00:24:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816355+0000", 2020-03-09T00:24:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.495e-05 2020-03-09T00:24:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816380+0000", 2020-03-09T00:24:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8318e-05 2020-03-09T00:24:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816399+0000", 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2422000000000001e-05 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816411+0000", 2020-03-09T00:24:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026289500000000002 2020-03-09T00:24:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816674+0000", 2020-03-09T00:24:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016398500000000001 2020-03-09T00:24:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816838+0000", 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056926199999999996 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.es1 38/32 ECSubRead(tid=29, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.814710+0000", 2020-03-09T00:24:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5248677900000001, 2020-03-09T00:24:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001428056, 2020-03-09T00:24:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814710+0000", 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814710+0000", 2020-03-09T00:24:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0417999999999998e-05 2020-03-09T00:24:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814751+0000", 2020-03-09T00:24:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3341e-05 2020-03-09T00:24:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814764+0000", 2020-03-09T00:24:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7047000000000001e-05 2020-03-09T00:24:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814781+0000", 2020-03-09T00:24:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7639999999999994e-06 2020-03-09T00:24:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814790+0000", 2020-03-09T00:24:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4024999999999997e-05 2020-03-09T00:24:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.814864+0000", 2020-03-09T00:24:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001274461 2020-03-09T00:24:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816138+0000", 2020-03-09T00:24:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001428056 2020-03-09T00:24:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.es1 38/32 ECSubRead(tid=30, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.815330+0000", 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5242480369999996, 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00109621, 2020-03-09T00:24:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815330+0000", 2020-03-09T00:24:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815330+0000", 2020-03-09T00:24:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6447999999999999e-05 2020-03-09T00:24:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815387+0000", 2020-03-09T00:24:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9862000000000001e-05 2020-03-09T00:24:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815406+0000", 2020-03-09T00:24:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0649000000000002e-05 2020-03-09T00:24:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815427+0000", 2020-03-09T00:24:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0011e-05 2020-03-09T00:24:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815437+0000", 2020-03-09T00:24:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075100099999999995 2020-03-09T00:24:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816188+0000", 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023823900000000001 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816426+0000", 2020-03-09T00:24:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00109621 2020-03-09T00:24:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.es1 38/32 ECSubRead(tid=31, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.815822+0000", 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5237566960000004, 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089488299999999996, 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815822+0000", 2020-03-09T00:24:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815822+0000", 2020-03-09T00:24:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7249999999999995e-05 2020-03-09T00:24:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815889+0000", 2020-03-09T00:24:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3901e-05 2020-03-09T00:24:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815903+0000", 2020-03-09T00:24:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9167000000000001e-05 2020-03-09T00:24:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815922+0000", 2020-03-09T00:24:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2190000000000004e-06 2020-03-09T00:24:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.815930+0000", 2020-03-09T00:24:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00053427100000000001 2020-03-09T00:24:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816464+0000", 2020-03-09T00:24:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025207499999999999 2020-03-09T00:24:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.816716+0000", 2020-03-09T00:24:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089488299999999996 2020-03-09T00:24:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 38/32 ECSubWrite(tid=32, reqid=client.4751.0:187, at_version=38'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.817656+0000", 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5219218970000004, 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023868520000000001, 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817656+0000", 2020-03-09T00:24:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817656+0000", 2020-03-09T00:24:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5086999999999998e-05 2020-03-09T00:24:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817691+0000", 2020-03-09T00:24:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9993e-05 2020-03-09T00:24:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817711+0000", 2020-03-09T00:24:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2850000000000001e-05 2020-03-09T00:24:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817724+0000", 2020-03-09T00:24:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2249999999999997e-06 2020-03-09T00:24:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817730+0000", 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6903000000000001e-05 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817777+0000", 2020-03-09T00:24:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2105e-05 2020-03-09T00:24:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817789+0000", 2020-03-09T00:24:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021889380000000001 2020-03-09T00:24:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.819978+0000", 2020-03-09T00:24:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4751000000000004e-05 2020-03-09T00:24:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820043+0000", 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023868520000000001 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpRead(3.1s2 38/32 ECSubRead(tid=24, to_read={3:8a6d1775:::smithi10428818-12:head=0,49152,0}, subchunks={3:8a6d1775:::smithi10428818-12:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.817928+0000", 2020-03-09T00:24:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5216498539999996, 2020-03-09T00:24:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041621799999999999, 2020-03-09T00:24:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817928+0000", 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817928+0000", 2020-03-09T00:24:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1074999999999994e-05 2020-03-09T00:24:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.817989+0000", 2020-03-09T00:24:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2061e-05 2020-03-09T00:24:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818002+0000", 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8643000000000001e-05 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818020+0000", 2020-03-09T00:24:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7940000000000001e-05 2020-03-09T00:24:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818038+0000", 2020-03-09T00:24:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118956 2020-03-09T00:24:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818157+0000", 2020-03-09T00:24:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000187543 2020-03-09T00:24:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818345+0000", 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041621799999999999 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 38/32 ECSubWrite(tid=33, reqid=client.4751.0:188, at_version=38'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.818499+0000", 2020-03-09T00:24:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5210788610000003, 2020-03-09T00:24:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002051935, 2020-03-09T00:24:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818499+0000", 2020-03-09T00:24:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818499+0000", 2020-03-09T00:24:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7651000000000001e-05 2020-03-09T00:24:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818557+0000", 2020-03-09T00:24:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5099000000000001e-05 2020-03-09T00:24:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818582+0000", 2020-03-09T00:24:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7297000000000001e-05 2020-03-09T00:24:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818599+0000", 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1230000000000002e-06 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818607+0000", 2020-03-09T00:24:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1027999999999999e-05 2020-03-09T00:24:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818668+0000", 2020-03-09T00:24:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7330000000000003e-06 2020-03-09T00:24:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.818675+0000", 2020-03-09T00:24:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001815847 2020-03-09T00:24:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820491+0000", 2020-03-09T00:24:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0157e-05 2020-03-09T00:24:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820551+0000", 2020-03-09T00:24:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002051935 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 38/32 ECSubWrite(tid=34, reqid=client.4751.0:189, at_version=38'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.820111+0000", 2020-03-09T00:24:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5194674150000003, 2020-03-09T00:24:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060961499999999998, 2020-03-09T00:24:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820111+0000", 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820111+0000", 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8763999999999993e-05 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820190+0000", 2020-03-09T00:24:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2374e-05 2020-03-09T00:24:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820212+0000", 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4408000000000001e-05 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820236+0000", 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8909999999999995e-06 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820244+0000", 2020-03-09T00:24:52.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4554000000000001e-05 2020-03-09T00:24:52.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820289+0000", 2020-03-09T00:24:52.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3076e-05 2020-03-09T00:24:52.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820302+0000", 2020-03-09T00:24:52.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036363699999999999 2020-03-09T00:24:52.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820666+0000", 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4911000000000003e-05 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.820720+0000", 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060961499999999998 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 38/32 ECSubWrite(tid=35, reqid=client.4751.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:52.444 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.825716+0000", 2020-03-09T00:24:52.445 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5138624370000002, 2020-03-09T00:24:52.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055575799999999995, 2020-03-09T00:24:52.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.445 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825716+0000", 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825716+0000", 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2017000000000002e-05 2020-03-09T00:24:52.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825748+0000", 2020-03-09T00:24:52.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1848000000000001e-05 2020-03-09T00:24:52.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825760+0000", 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0003999999999999e-05 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825770+0000", 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2079999999999999e-06 2020-03-09T00:24:52.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825775+0000", 2020-03-09T00:24:52.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0890999999999998e-05 2020-03-09T00:24:52.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825826+0000", 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6707000000000001e-05 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825842+0000", 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039134100000000002 2020-03-09T00:24:52.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826234+0000", 2020-03-09T00:24:52.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7741999999999999e-05 2020-03-09T00:24:52.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.826272+0000", 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055575799999999995 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.452 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as3 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:24:52.453 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.864217+0000", 2020-03-09T00:24:52.453 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.4753613830000001, 2020-03-09T00:24:52.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038879940000000001, 2020-03-09T00:24:52.453 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.453 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.453 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864217+0000", 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864217+0000", 2020-03-09T00:24:52.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000153547 2020-03-09T00:24:52.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864370+0000", 2020-03-09T00:24:52.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1137000000000001e-05 2020-03-09T00:24:52.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864392+0000", 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3069e-05 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864405+0000", 2020-03-09T00:24:52.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.251e-06 2020-03-09T00:24:52.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864411+0000", 2020-03-09T00:24:52.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1211000000000001e-05 2020-03-09T00:24:52.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864462+0000", 2020-03-09T00:24:52.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.516e-06 2020-03-09T00:24:52.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864472+0000", 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00359319 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.868065+0000", 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0073000000000001e-05 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.868105+0000", 2020-03-09T00:24:52.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038879940000000001 2020-03-09T00:24:52.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.460 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as3 38/32 ECSubWrite(tid=6, reqid=client.4751.0:207, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.870753+0000", 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.4688256119999998, 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058428799999999997, 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.461 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870753+0000", 2020-03-09T00:24:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870753+0000", 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4719999999999998e-05 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870807+0000", 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5242000000000001e-05 2020-03-09T00:24:52.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870823+0000", 2020-03-09T00:24:52.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1016999999999999e-05 2020-03-09T00:24:52.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870844+0000", 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4490000000000002e-06 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870851+0000", 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9332000000000001e-05 2020-03-09T00:24:52.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870900+0000", 2020-03-09T00:24:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9270000000000005e-06 2020-03-09T00:24:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.870908+0000", 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039214099999999998 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.871300+0000", 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.646e-05 2020-03-09T00:24:52.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.871337+0000", 2020-03-09T00:24:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058428799999999997 2020-03-09T00:24:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.469 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs3 39/32 ECSubWrite(tid=22, reqid=client.4751.0:210, at_version=39'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:52.469 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:48.828466+0000", 2020-03-09T00:24:52.469 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.5111123829999999, 2020-03-09T00:24:52.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004661971, 2020-03-09T00:24:52.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.828466+0000", 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.828466+0000", 2020-03-09T00:24:52.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067655999999999999 2020-03-09T00:24:52.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829142+0000", 2020-03-09T00:24:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6149e-05 2020-03-09T00:24:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829169+0000", 2020-03-09T00:24:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6626e-05 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829185+0000", 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2470000000000002e-06 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829192+0000", 2020-03-09T00:24:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2005e-05 2020-03-09T00:24:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829254+0000", 2020-03-09T00:24:52.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5999e-05 2020-03-09T00:24:52.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829270+0000", 2020-03-09T00:24:52.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038224880000000002 2020-03-09T00:24:52.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.833093+0000", 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4897000000000001e-05 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.833128+0000", 2020-03-09T00:24:52.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004661971 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs3 39/32 ECSubWrite(tid=23, reqid=client.4751.0:210, at_version=0'0, trim_to=0'0, roll_forward_to=39'8))", 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:48.834239+0000", 2020-03-09T00:24:52.477 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.505339175, 2020-03-09T00:24:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072207499999999997, 2020-03-09T00:24:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834239+0000", 2020-03-09T00:24:52.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834239+0000", 2020-03-09T00:24:52.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1124999999999998e-05 2020-03-09T00:24:52.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834270+0000", 2020-03-09T00:24:52.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7660000000000004e-06 2020-03-09T00:24:52.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834279+0000", 2020-03-09T00:24:52.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1155000000000001e-05 2020-03-09T00:24:52.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834290+0000", 2020-03-09T00:24:52.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4299999999999999e-06 2020-03-09T00:24:52.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834295+0000", 2020-03-09T00:24:52.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1746999999999998e-05 2020-03-09T00:24:52.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834346+0000", 2020-03-09T00:24:52.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4321999999999999e-05 2020-03-09T00:24:52.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834361+0000", 2020-03-09T00:24:52.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056954399999999995 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834930+0000", 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0985999999999998e-05 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.834961+0000", 2020-03-09T00:24:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072207499999999997 2020-03-09T00:24:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s2 42/32 ECSubWrite(tid=32, reqid=client.4751.0:224, at_version=42'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:51.832914+0000", 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.50666416800000003, 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013472079999999999, 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832914+0000", 2020-03-09T00:24:52.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832914+0000", 2020-03-09T00:24:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5107000000000001e-05 2020-03-09T00:24:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832949+0000", 2020-03-09T00:24:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0100999999999999e-05 2020-03-09T00:24:52.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832969+0000", 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8753000000000002e-05 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832988+0000", 2020-03-09T00:24:52.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8050000000000001e-06 2020-03-09T00:24:52.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832998+0000", 2020-03-09T00:24:52.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7568e-05 2020-03-09T00:24:52.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.833075+0000", 2020-03-09T00:24:52.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8029999999999998e-05 2020-03-09T00:24:52.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.833093+0000", 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001108948 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.834202+0000", 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8896e-05 2020-03-09T00:24:52.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.834261+0000", 2020-03-09T00:24:52.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013472079999999999 2020-03-09T00:24:52.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s2 42/32 ECSubWrite(tid=33, reqid=client.4751.0:224, at_version=0'0, trim_to=0'0, roll_forward_to=42'6))", 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:51.835289+0000", 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.50428903199999997, 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067973100000000004, 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:52.496 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.497 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:52.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:52.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835289+0000", 2020-03-09T00:24:52.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:52.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:52.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835289+0000", 2020-03-09T00:24:52.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3338999999999997e-05 2020-03-09T00:24:52.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:52.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835363+0000", 2020-03-09T00:24:52.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8366e-05 2020-03-09T00:24:52.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835381+0000", 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8127000000000001e-05 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835399+0000", 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2439999999999999e-06 2020-03-09T00:24:52.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835408+0000", 2020-03-09T00:24:52.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0668999999999998e-05 2020-03-09T00:24:52.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835469+0000", 2020-03-09T00:24:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0080000000000001e-05 2020-03-09T00:24:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835489+0000", 2020-03-09T00:24:52.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043685399999999997 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835926+0000", 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3052e-05 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:52.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:52.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.835969+0000", 2020-03-09T00:24:52.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067973100000000004 2020-03-09T00:24:52.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:52.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:52.505 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:52.506 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:24:52.507 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:52.504+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:52.607 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:52.605+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:52.630 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:52.630 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:52.630 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:24:52.630 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:52.638 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:24:52.707 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:52.706+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:52.794 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:52.794 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:52.795 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:24:52.795 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:24:52.795 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:52.802 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:24:52.836 INFO:tasks.rados.rados.0.smithi104.stdout:85: done (4 left) 2020-03-09T00:24:52.836 INFO:tasks.rados.rados.0.smithi104.stdout:86: done (3 left) 2020-03-09T00:24:52.836 INFO:tasks.rados.rados.0.smithi104.stdout:87: done (2 left) 2020-03-09T00:24:52.836 INFO:tasks.rados.rados.0.smithi104.stdout:88: done (1 left) 2020-03-09T00:24:52.836 INFO:tasks.rados.rados.0.smithi104.stdout:89: done (0 left) 2020-03-09T00:24:52.836 INFO:tasks.rados.rados.0.smithi104.stdout:90: delete oid 15 current snap is 6 2020-03-09T00:24:52.841 INFO:tasks.rados.rados.0.smithi104.stdout:90: done (0 left) 2020-03-09T00:24:52.841 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 18 current snap is 6 2020-03-09T00:24:52.842 INFO:tasks.rados.rados.0.smithi104.stdout:91: seq_num 59 ranges {294912=622592,917504=376832} 2020-03-09T00:24:52.846 INFO:tasks.rados.rados.0.smithi104.stdout:91: writing smithi10428818-18 from 294912 to 917504 tid 1 2020-03-09T00:24:52.847 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:52.846+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:52.848 INFO:tasks.rados.rados.0.smithi104.stdout:91: writing smithi10428818-18 from 917504 to 1294336 tid 2 2020-03-09T00:24:52.848 INFO:tasks.rados.rados.0.smithi104.stdout:92: snap_remove snap 4 2020-03-09T00:24:52.909 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:52.908+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:52.922 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:52.922 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:24:52.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:24:52.923 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:24:52.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.923 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:184 3.es0 3:77c8bc20:::smithi10428818-47:4 [read 0~786432,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:52.923 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.811675+0000", 2020-03-09T00:24:52.923 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1083499510000001, 2020-03-09T00:24:52.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014888533000000001, 2020-03-09T00:24:52.924 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:52.924 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:24:52.924 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:24:52.924 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:24:52.924 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:52.924 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 184 2020-03-09T00:24:52.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811675+0000", 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:52.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811675+0000", 2020-03-09T00:24:52.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2947000000000001e-05 2020-03-09T00:24:52.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:52.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811717+0000", 2020-03-09T00:24:52.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4289999999999998e-06 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811724+0000", 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1924e-05 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:52.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811736+0000", 2020-03-09T00:24:52.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3980000000000002e-06 2020-03-09T00:24:52.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811743+0000", 2020-03-09T00:24:52.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0408999999999998e-05 2020-03-09T00:24:52.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811794+0000", 2020-03-09T00:24:52.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2080000000000003e-06 2020-03-09T00:24:52.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:24:52.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811801+0000", 2020-03-09T00:24:52.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021651610000000001 2020-03-09T00:24:52.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.813966+0000", 2020-03-09T00:24:52.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000129247 2020-03-09T00:24:52.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:52.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814095+0000", 2020-03-09T00:24:52.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012467809999999999 2020-03-09T00:24:52.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:52.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.826563+0000", 2020-03-09T00:24:52.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014888533000000001 2020-03-09T00:24:52.932 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.932 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:52.933 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.933 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:185 3.es0 3:77c8bc20:::smithi10428818-47:4 [read 0~786432,checksum] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:52.933 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.811782+0000", 2020-03-09T00:24:52.933 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1082424499999997, 2020-03-09T00:24:52.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015338732000000001, 2020-03-09T00:24:52.934 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:52.934 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:24:52.934 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:24:52.934 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:24:52.934 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:52.935 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 185 2020-03-09T00:24:52.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.935 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:52.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:52.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811782+0000", 2020-03-09T00:24:52.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:52.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811782+0000", 2020-03-09T00:24:52.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7703999999999997e-05 2020-03-09T00:24:52.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811820+0000", 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5599999999999998e-06 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811823+0000", 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3010000000000003e-06 2020-03-09T00:24:52.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:52.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811830+0000", 2020-03-09T00:24:52.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9929999999999997e-06 2020-03-09T00:24:52.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811834+0000", 2020-03-09T00:24:52.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0899000000000003e-05 2020-03-09T00:24:52.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811874+0000", 2020-03-09T00:24:52.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9910000000000008e-06 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811883+0000", 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028042219999999999 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814688+0000", 2020-03-09T00:24:52.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3833999999999998e-05 2020-03-09T00:24:52.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:52.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814762+0000", 2020-03-09T00:24:52.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012359228 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.827121+0000", 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015338732000000001 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:52.944 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.946 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:186 3.es0 3:77c8bc20:::smithi10428818-47:4 [read 0~786432,checksum] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:52.946 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.811886+0000", 2020-03-09T00:24:52.946 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1081387789999999, 2020-03-09T00:24:52.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015779517999999999, 2020-03-09T00:24:52.947 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:52.947 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:24:52.947 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:24:52.947 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:24:52.947 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:52.947 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 186 2020-03-09T00:24:52.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.948 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:52.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:52.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811886+0000", 2020-03-09T00:24:52.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811886+0000", 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4892000000000001e-05 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811951+0000", 2020-03-09T00:24:52.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5570000000000001e-06 2020-03-09T00:24:52.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:52.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811957+0000", 2020-03-09T00:24:52.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2e-05 2020-03-09T00:24:52.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811969+0000", 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7720000000000002e-06 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811974+0000", 2020-03-09T00:24:52.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3767999999999998e-05 2020-03-09T00:24:52.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812018+0000", 2020-03-09T00:24:52.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5399999999999996e-06 2020-03-09T00:24:52.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:52.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812026+0000", 2020-03-09T00:24:52.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003166079 2020-03-09T00:24:52.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815192+0000", 2020-03-09T00:24:52.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000126334 2020-03-09T00:24:52.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:52.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815319+0000", 2020-03-09T00:24:52.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012346576 2020-03-09T00:24:52.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:52.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.827665+0000", 2020-03-09T00:24:52.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015779517999999999 2020-03-09T00:24:52.955 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.955 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:52.955 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:187 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.812020+0000", 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1080042470000002, 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012768409999999999, 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:52.956 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:52.957 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:24:52.957 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:24:52.957 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:52.957 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 187 2020-03-09T00:24:52.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.957 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:52.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:52.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812020+0000", 2020-03-09T00:24:52.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:52.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812020+0000", 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9535000000000001e-05 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812050+0000", 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3790000000000003e-06 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:52.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812056+0000", 2020-03-09T00:24:52.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3845999999999999e-05 2020-03-09T00:24:52.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:52.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812070+0000", 2020-03-09T00:24:52.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9150000000000004e-06 2020-03-09T00:24:52.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812075+0000", 2020-03-09T00:24:52.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000284436 2020-03-09T00:24:52.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812359+0000", 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9709999999999993e-06 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:52.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812368+0000", 2020-03-09T00:24:52.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035513720000000001 2020-03-09T00:24:52.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815920+0000", 2020-03-09T00:24:52.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000108948 2020-03-09T00:24:52.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816029+0000", 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00091191999999999998 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816941+0000", 2020-03-09T00:24:52.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031524320000000002 2020-03-09T00:24:52.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.820093+0000", 2020-03-09T00:24:52.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046122180000000004 2020-03-09T00:24:52.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.824705+0000", 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3437999999999996e-05 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:52.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.824789+0000", 2020-03-09T00:24:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012768409999999999 2020-03-09T00:24:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.967 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:188 3.es0 3:73e16f4f:::smithi10428818-13:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.812124+0000", 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1079009040000001, 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012867518, 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:24:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:52.969 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 188 2020-03-09T00:24:52.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.969 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:52.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:52.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812124+0000", 2020-03-09T00:24:52.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812124+0000", 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5053999999999999e-05 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:52.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812169+0000", 2020-03-09T00:24:52.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9200000000000001e-06 2020-03-09T00:24:52.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:52.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812175+0000", 2020-03-09T00:24:52.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1307e-05 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812186+0000", 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.293e-06 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812191+0000", 2020-03-09T00:24:52.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019990900000000001 2020-03-09T00:24:52.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812391+0000", 2020-03-09T00:24:52.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0049999999999998e-06 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812398+0000", 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005077949 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817476+0000", 2020-03-09T00:24:52.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000110153 2020-03-09T00:24:52.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:52.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817586+0000", 2020-03-09T00:24:52.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00074894100000000004 2020-03-09T00:24:52.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818335+0000", 2020-03-09T00:24:52.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048146070000000003 2020-03-09T00:24:52.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823150+0000", 2020-03-09T00:24:52.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017606410000000001 2020-03-09T00:24:52.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:24:52.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.824910+0000", 2020-03-09T00:24:52.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0739000000000001e-05 2020-03-09T00:24:52.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:52.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.824991+0000", 2020-03-09T00:24:52.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012867518 2020-03-09T00:24:52.979 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.979 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:52.979 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:189 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.812236+0000", 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1077883770000003, 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01308689, 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:24:52.980 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:24:52.981 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:52.981 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 189 2020-03-09T00:24:52.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.981 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:52.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:52.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812236+0000", 2020-03-09T00:24:52.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:52.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812236+0000", 2020-03-09T00:24:52.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5000000000000003e-05 2020-03-09T00:24:52.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812281+0000", 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1430000000000001e-06 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:52.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812288+0000", 2020-03-09T00:24:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1058e-05 2020-03-09T00:24:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812299+0000", 2020-03-09T00:24:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1740000000000002e-06 2020-03-09T00:24:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812303+0000", 2020-03-09T00:24:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116718 2020-03-09T00:24:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812420+0000", 2020-03-09T00:24:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0929999999999997e-06 2020-03-09T00:24:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812427+0000", 2020-03-09T00:24:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0063277869999999996 2020-03-09T00:24:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818755+0000", 2020-03-09T00:24:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000111463 2020-03-09T00:24:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:24:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818867+0000", 2020-03-09T00:24:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010359799999999999 2020-03-09T00:24:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819902+0000", 2020-03-09T00:24:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032516210000000001 2020-03-09T00:24:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823154+0000", 2020-03-09T00:24:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020912859999999999 2020-03-09T00:24:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.825245+0000", 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7566999999999998e-05 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.825323+0000", 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01308689 2020-03-09T00:24:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpRead(3.es0 38/32 ECSubRead(tid=29, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.814472+0000", 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1055525770000001, 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084433129999999992, 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814472+0000", 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814472+0000", 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814472+0000", 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814472+0000", 2020-03-09T00:24:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7573000000000001e-05 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814509+0000", 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2359999999999998e-06 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814517+0000", 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056168920000000001 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.820134+0000", 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027816120000000001 2020-03-09T00:24:52.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.822915+0000", 2020-03-09T00:24:53.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084433129999999992 2020-03-09T00:24:53.000 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.000 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.000 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.001 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpRead(3.es0 38/32 ECSubRead(tid=30, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:53.001 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.815010+0000", 2020-03-09T00:24:53.001 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1050148350000004, 2020-03-09T00:24:53.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081456949999999997, 2020-03-09T00:24:53.001 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.002 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.002 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815010+0000", 2020-03-09T00:24:53.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815010+0000", 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815010+0000", 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815010+0000", 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9915999999999997e-05 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815110+0000", 2020-03-09T00:24:53.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7989999999999996e-06 2020-03-09T00:24:53.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815116+0000", 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078574140000000001 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.822974+0000", 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000181566 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823155+0000", 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081456949999999997 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpRead(3.es0 38/32 ECSubRead(tid=31, to_read={3:77c8bc20:::smithi10428818-47:head=0,196608,0}, subchunks={3:77c8bc20:::smithi10428818-47:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.815613+0000", 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.104411925, 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077765430000000003, 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815613+0000", 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815613+0000", 2020-03-09T00:24:53.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815613+0000", 2020-03-09T00:24:53.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815613+0000", 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016908299999999999 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815782+0000", 2020-03-09T00:24:53.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9750000000000001e-06 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815789+0000", 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074067559999999996 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823195+0000", 2020-03-09T00:24:53.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019372899999999999 2020-03-09T00:24:53.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823389+0000", 2020-03-09T00:24:53.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077765430000000003 2020-03-09T00:24:53.014 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.014 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.014 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.014 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 38/32 ECSubReadReply(tid=29, attrs_read=0))", 2020-03-09T00:24:53.015 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.815888+0000", 2020-03-09T00:24:53.015 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1041360359999999, 2020-03-09T00:24:53.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007599059, 2020-03-09T00:24:53.015 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.015 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.015 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815888+0000", 2020-03-09T00:24:53.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815888+0000", 2020-03-09T00:24:53.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050610200000000003 2020-03-09T00:24:53.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816395+0000", 2020-03-09T00:24:53.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1517000000000001e-05 2020-03-09T00:24:53.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816416+0000", 2020-03-09T00:24:53.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3338e-05 2020-03-09T00:24:53.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816429+0000", 2020-03-09T00:24:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.417e-05 2020-03-09T00:24:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816444+0000", 2020-03-09T00:24:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0069874359999999996 2020-03-09T00:24:53.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823431+0000", 2020-03-09T00:24:53.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6496000000000003e-05 2020-03-09T00:24:53.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823487+0000", 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007599059 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.022 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 37/32 ECSubReadReply(tid=29, attrs_read=0))", 2020-03-09T00:24:53.022 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.816111+0000", 2020-03-09T00:24:53.022 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.103912974, 2020-03-09T00:24:53.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074451600000000001, 2020-03-09T00:24:53.022 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.022 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816111+0000", 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816111+0000", 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00063791300000000004 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816749+0000", 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0013000000000001e-05 2020-03-09T00:24:53.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816759+0000", 2020-03-09T00:24:53.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0579000000000001e-05 2020-03-09T00:24:53.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816780+0000", 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2070000000000003e-06 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816785+0000", 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067325869999999999 2020-03-09T00:24:53.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823518+0000", 2020-03-09T00:24:53.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8860999999999999e-05 2020-03-09T00:24:53.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823557+0000", 2020-03-09T00:24:53.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074451600000000001 2020-03-09T00:24:53.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.028 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 38/32 ECSubReadReply(tid=30, attrs_read=0))", 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.816488+0000", 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1035365029999999, 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072921180000000002, 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.029 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.030 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816488+0000", 2020-03-09T00:24:53.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816488+0000", 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013682169999999999 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817856+0000", 2020-03-09T00:24:53.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.098e-05 2020-03-09T00:24:53.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817867+0000", 2020-03-09T00:24:53.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2194999999999999e-05 2020-03-09T00:24:53.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817879+0000", 2020-03-09T00:24:53.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8130000000000003e-06 2020-03-09T00:24:53.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817884+0000", 2020-03-09T00:24:53.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005856867 2020-03-09T00:24:53.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823741+0000", 2020-03-09T00:24:53.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9045999999999999e-05 2020-03-09T00:24:53.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823780+0000", 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072921180000000002 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 37/32 ECSubReadReply(tid=30, attrs_read=0))", 2020-03-09T00:24:53.036 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.816889+0000", 2020-03-09T00:24:53.037 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1031354779999996, 2020-03-09T00:24:53.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067068830000000003, 2020-03-09T00:24:53.037 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.037 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.037 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816889+0000", 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816889+0000", 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00025456699999999998 2020-03-09T00:24:53.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817144+0000", 2020-03-09T00:24:53.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1687e-05 2020-03-09T00:24:53.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817155+0000", 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4134000000000001e-05 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817169+0000", 2020-03-09T00:24:53.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8250000000000001e-06 2020-03-09T00:24:53.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817175+0000", 2020-03-09T00:24:53.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0063956760000000003 2020-03-09T00:24:53.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823571+0000", 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4994000000000002e-05 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823596+0000", 2020-03-09T00:24:53.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067068830000000003 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 37/32 ECSubReadReply(tid=31, attrs_read=0))", 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.817228+0000", 2020-03-09T00:24:53.043 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1027963620000003, 2020-03-09T00:24:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0064499020000000004, 2020-03-09T00:24:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817228+0000", 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817228+0000", 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00023996 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817468+0000", 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6469999999999993e-06 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817478+0000", 2020-03-09T00:24:53.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1399999999999999e-05 2020-03-09T00:24:53.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817489+0000", 2020-03-09T00:24:53.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7419999999999997e-06 2020-03-09T00:24:53.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817494+0000", 2020-03-09T00:24:53.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0061395809999999999 2020-03-09T00:24:53.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823633+0000", 2020-03-09T00:24:53.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4571999999999999e-05 2020-03-09T00:24:53.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823678+0000", 2020-03-09T00:24:53.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0064499020000000004 2020-03-09T00:24:53.049 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 37/32 ECSubReadReply(tid=29, attrs_read=0))", 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.817263+0000", 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1027611840000002, 2020-03-09T00:24:53.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0064503729999999997, 2020-03-09T00:24:53.051 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.051 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.051 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817263+0000", 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817263+0000", 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000483442 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817747+0000", 2020-03-09T00:24:53.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.664e-05 2020-03-09T00:24:53.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817763+0000", 2020-03-09T00:24:53.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8970999999999998e-05 2020-03-09T00:24:53.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817782+0000", 2020-03-09T00:24:53.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2559999999999999e-06 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817789+0000", 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059019349999999996 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823691+0000", 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3129000000000002e-05 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823714+0000", 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0064503729999999997 2020-03-09T00:24:53.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 37/32 ECSubReadReply(tid=30, attrs_read=0))", 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.817866+0000", 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1021586330000002, 2020-03-09T00:24:53.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059480430000000001, 2020-03-09T00:24:53.058 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.058 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.058 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817866+0000", 2020-03-09T00:24:53.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817866+0000", 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00024917400000000001 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818115+0000", 2020-03-09T00:24:53.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1179999999999998e-06 2020-03-09T00:24:53.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818124+0000", 2020-03-09T00:24:53.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.184e-05 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818136+0000", 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3832000000000001e-05 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818150+0000", 2020-03-09T00:24:53.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056409549999999996 2020-03-09T00:24:53.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823791+0000", 2020-03-09T00:24:53.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3124000000000001e-05 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823814+0000", 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059480430000000001 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.063 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.064 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 38/32 ECSubReadReply(tid=31, attrs_read=0))", 2020-03-09T00:24:53.065 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.817959+0000", 2020-03-09T00:24:53.065 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1020658040000004, 2020-03-09T00:24:53.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005920687, 2020-03-09T00:24:53.066 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.066 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.066 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817959+0000", 2020-03-09T00:24:53.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.817959+0000", 2020-03-09T00:24:53.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00020193300000000001 2020-03-09T00:24:53.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818161+0000", 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7030000000000001e-06 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818170+0000", 2020-03-09T00:24:53.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3095000000000001e-05 2020-03-09T00:24:53.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818183+0000", 2020-03-09T00:24:53.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9199999999999997e-06 2020-03-09T00:24:53.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818187+0000", 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056550089999999999 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823842+0000", 2020-03-09T00:24:53.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7027000000000003e-05 2020-03-09T00:24:53.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823879+0000", 2020-03-09T00:24:53.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005920687 2020-03-09T00:24:53.071 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpReadReply(3.es0 37/32 ECSubReadReply(tid=31, attrs_read=0))", 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.818204+0000", 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1018204190000001, 2020-03-09T00:24:53.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058620019999999998, 2020-03-09T00:24:53.073 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.073 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.074 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818204+0000", 2020-03-09T00:24:53.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818204+0000", 2020-03-09T00:24:53.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015242890000000001 2020-03-09T00:24:53.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819728+0000", 2020-03-09T00:24:53.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0635000000000001e-05 2020-03-09T00:24:53.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819739+0000", 2020-03-09T00:24:53.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3326000000000001e-05 2020-03-09T00:24:53.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819752+0000", 2020-03-09T00:24:53.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0839999999999999e-06 2020-03-09T00:24:53.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819757+0000", 2020-03-09T00:24:53.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042545810000000003 2020-03-09T00:24:53.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.824012+0000", 2020-03-09T00:24:53.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4086999999999999e-05 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.824066+0000", 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058620019999999998 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.080 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.es0 38/32 ECSubWriteReply(tid=32, last_complete=38'12, committed=1, applied=1))", 2020-03-09T00:24:53.080 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.819374+0000", 2020-03-09T00:24:53.080 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.1006506610000004, 2020-03-09T00:24:53.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045420640000000002, 2020-03-09T00:24:53.080 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819374+0000", 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819374+0000", 2020-03-09T00:24:53.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1631999999999995e-05 2020-03-09T00:24:53.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819435+0000", 2020-03-09T00:24:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4360000000000002e-06 2020-03-09T00:24:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819444+0000", 2020-03-09T00:24:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6234000000000001e-05 2020-03-09T00:24:53.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819460+0000", 2020-03-09T00:24:53.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4309999999999997e-06 2020-03-09T00:24:53.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.819468+0000", 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044382639999999999 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823906+0000", 2020-03-09T00:24:53.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0067e-05 2020-03-09T00:24:53.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.823916+0000", 2020-03-09T00:24:53.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045420640000000002 2020-03-09T00:24:53.086 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.086 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.087 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.087 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.5s6 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:53.087 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.830429+0000", 2020-03-09T00:24:53.087 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.0895958229999998, 2020-03-09T00:24:53.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045477989999999999, 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830429+0000", 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:53.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:53.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830429+0000", 2020-03-09T00:24:53.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00034433800000000002 2020-03-09T00:24:53.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:53.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830773+0000", 2020-03-09T00:24:53.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7801000000000001e-05 2020-03-09T00:24:53.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:53.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830791+0000", 2020-03-09T00:24:53.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0618000000000001e-05 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830801+0000", 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8439999999999997e-06 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.092 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830806+0000", 2020-03-09T00:24:53.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0352000000000001e-05 2020-03-09T00:24:53.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:53.092 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830867+0000", 2020-03-09T00:24:53.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4986e-05 2020-03-09T00:24:53.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.830882+0000", 2020-03-09T00:24:53.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00405741 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.834939+0000", 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7450000000000002e-05 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:53.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.834976+0000", 2020-03-09T00:24:53.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045477989999999999 2020-03-09T00:24:53.095 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.095 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.095 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.095 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:53.095 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:53.096 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:53.097 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:53.095+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:53.097 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:24:53.197 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:53.194+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:53.224 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:53.225 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:53.225 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:24:53.225 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:53.233 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:24:53.317 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:53.315+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:53.388 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:53.389 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:53.389 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:24:53.389 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:24:53.389 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:53.397 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:24:53.399 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:53.398+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:24:53.500 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:53.499+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:53.553 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:53.553 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:24:53.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:24:53.553 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:24:53.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.553 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:181 3.fs0 3:f1702e73:::smithi10428818-39:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:53.554 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811361+0000", 2020-03-09T00:24:53.554 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7388655100000001, 2020-03-09T00:24:53.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033103770000000002, 2020-03-09T00:24:53.554 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.554 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:24:53.554 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:53.554 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 181 2020-03-09T00:24:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811361+0000", 2020-03-09T00:24:53.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811361+0000", 2020-03-09T00:24:53.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0516e-05 2020-03-09T00:24:53.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811442+0000", 2020-03-09T00:24:53.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4930000000000004e-06 2020-03-09T00:24:53.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811451+0000", 2020-03-09T00:24:53.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4593000000000001e-05 2020-03-09T00:24:53.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811466+0000", 2020-03-09T00:24:53.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3470000000000001e-06 2020-03-09T00:24:53.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811473+0000", 2020-03-09T00:24:53.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000101542 2020-03-09T00:24:53.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811575+0000", 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108311 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811683+0000", 2020-03-09T00:24:53.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029885749999999998 2020-03-09T00:24:53.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.814672+0000", 2020-03-09T00:24:53.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033103770000000002 2020-03-09T00:24:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.562 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:182 3.fs0 3:f1702e73:::smithi10428818-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:53.563 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811530+0000", 2020-03-09T00:24:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7386966170000004, 2020-03-09T00:24:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036443420000000001, 2020-03-09T00:24:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:24:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:53.564 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 182 2020-03-09T00:24:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811530+0000", 2020-03-09T00:24:53.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811530+0000", 2020-03-09T00:24:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7975000000000001e-05 2020-03-09T00:24:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811578+0000", 2020-03-09T00:24:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1280000000000001e-06 2020-03-09T00:24:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811582+0000", 2020-03-09T00:24:53.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3300000000000001e-06 2020-03-09T00:24:53.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811589+0000", 2020-03-09T00:24:53.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.36e-06 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811593+0000", 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056580499999999995 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.812159+0000", 2020-03-09T00:24:53.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011756699999999999 2020-03-09T00:24:53.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.812276+0000", 2020-03-09T00:24:53.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002898177 2020-03-09T00:24:53.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.815174+0000", 2020-03-09T00:24:53.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036443420000000001 2020-03-09T00:24:53.573 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.573 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.573 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.573 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:183 3.fs0 3:f1702e73:::smithi10428818-39:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:53.573 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811625+0000", 2020-03-09T00:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7386016030000002, 2020-03-09T00:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010065457999999999, 2020-03-09T00:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:53.574 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:53.575 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 183 2020-03-09T00:24:53.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.575 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811625+0000", 2020-03-09T00:24:53.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811625+0000", 2020-03-09T00:24:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1186999999999997e-05 2020-03-09T00:24:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811666+0000", 2020-03-09T00:24:53.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.287e-06 2020-03-09T00:24:53.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811670+0000", 2020-03-09T00:24:53.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2339999999999999e-06 2020-03-09T00:24:53.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811676+0000", 2020-03-09T00:24:53.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6579999999999998e-06 2020-03-09T00:24:53.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811678+0000", 2020-03-09T00:24:53.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003540821 2020-03-09T00:24:53.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.815219+0000", 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000116397 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.815336+0000", 2020-03-09T00:24:53.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063548739999999999 2020-03-09T00:24:53.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.821691+0000", 2020-03-09T00:24:53.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010065457999999999 2020-03-09T00:24:53.581 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.581 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.582 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.582 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:192 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:53.582 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811713+0000", 2020-03-09T00:24:53.582 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7385140200000002, 2020-03-09T00:24:53.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092082169999999994, 2020-03-09T00:24:53.583 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.583 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:53.583 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:53.583 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:53.583 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:53.583 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 192 2020-03-09T00:24:53.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.584 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811713+0000", 2020-03-09T00:24:53.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811713+0000", 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3635999999999999e-05 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811736+0000", 2020-03-09T00:24:53.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2480000000000001e-06 2020-03-09T00:24:53.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811740+0000", 2020-03-09T00:24:53.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4670000000000001e-06 2020-03-09T00:24:53.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811746+0000", 2020-03-09T00:24:53.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7209999999999999e-06 2020-03-09T00:24:53.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811749+0000", 2020-03-09T00:24:53.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039885349999999997 2020-03-09T00:24:53.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.815737+0000", 2020-03-09T00:24:53.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104991 2020-03-09T00:24:53.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.815842+0000", 2020-03-09T00:24:53.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089497600000000002 2020-03-09T00:24:53.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.816737+0000", 2020-03-09T00:24:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00098868000000000003 2020-03-09T00:24:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817726+0000", 2020-03-09T00:24:53.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031448270000000002 2020-03-09T00:24:53.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:24:53.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820871+0000", 2020-03-09T00:24:53.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0136000000000002e-05 2020-03-09T00:24:53.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820921+0000", 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092082169999999994 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:193 3.fs0 3:fa16d489:::smithi10428818-27:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811772+0000", 2020-03-09T00:24:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7384550790000004, 2020-03-09T00:24:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012182698, 2020-03-09T00:24:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 193 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811772+0000", 2020-03-09T00:24:53.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811772+0000", 2020-03-09T00:24:53.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8804000000000001e-05 2020-03-09T00:24:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811800+0000", 2020-03-09T00:24:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0749999999999997e-06 2020-03-09T00:24:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811805+0000", 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1240000000000002e-06 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811812+0000", 2020-03-09T00:24:53.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9950000000000001e-06 2020-03-09T00:24:53.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811815+0000", 2020-03-09T00:24:53.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059357740000000004 2020-03-09T00:24:53.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817750+0000", 2020-03-09T00:24:53.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111483 2020-03-09T00:24:53.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817862+0000", 2020-03-09T00:24:53.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00073629399999999997 2020-03-09T00:24:53.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818598+0000", 2020-03-09T00:24:53.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001344425 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819943+0000", 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003966832 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:24:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823909+0000", 2020-03-09T00:24:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4892000000000003e-05 2020-03-09T00:24:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823954+0000", 2020-03-09T00:24:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012182698 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:194 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811838+0000", 2020-03-09T00:24:53.603 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7383881859999999, 2020-03-09T00:24:53.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012302923, 2020-03-09T00:24:53.604 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.604 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:53.604 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:53.604 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:53.604 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:53.604 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 194 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811838+0000", 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811838+0000", 2020-03-09T00:24:53.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3748999999999999e-05 2020-03-09T00:24:53.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811862+0000", 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.455e-06 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811866+0000", 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4569999999999999e-06 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811873+0000", 2020-03-09T00:24:53.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7450000000000001e-06 2020-03-09T00:24:53.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811877+0000", 2020-03-09T00:24:53.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007146693 2020-03-09T00:24:53.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819024+0000", 2020-03-09T00:24:53.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104384 2020-03-09T00:24:53.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819128+0000", 2020-03-09T00:24:53.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052495999999999999 2020-03-09T00:24:53.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819653+0000", 2020-03-09T00:24:53.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000337096 2020-03-09T00:24:53.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819990+0000", 2020-03-09T00:24:53.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040765749999999998 2020-03-09T00:24:53.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:24:53.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.824067+0000", 2020-03-09T00:24:53.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4808999999999995e-05 2020-03-09T00:24:53.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.824141+0000", 2020-03-09T00:24:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012302923 2020-03-09T00:24:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.616 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.4s5 38/32 ECSubWrite(tid=20, reqid=client.4751.0:190, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:24:53.616 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.823183+0000", 2020-03-09T00:24:53.616 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7270432810000003, 2020-03-09T00:24:53.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021535149999999999, 2020-03-09T00:24:53.618 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.619 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.619 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823183+0000", 2020-03-09T00:24:53.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823183+0000", 2020-03-09T00:24:53.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3707000000000003e-05 2020-03-09T00:24:53.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823277+0000", 2020-03-09T00:24:53.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5335999999999999e-05 2020-03-09T00:24:53.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823302+0000", 2020-03-09T00:24:53.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8788999999999999e-05 2020-03-09T00:24:53.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823321+0000", 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.435e-06 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823329+0000", 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8507000000000002e-05 2020-03-09T00:24:53.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823367+0000", 2020-03-09T00:24:53.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3139999999999998e-06 2020-03-09T00:24:53.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823374+0000", 2020-03-09T00:24:53.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019188219999999999 2020-03-09T00:24:53.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825293+0000", 2020-03-09T00:24:53.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3605e-05 2020-03-09T00:24:53.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825337+0000", 2020-03-09T00:24:53.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021535149999999999 2020-03-09T00:24:53.628 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.628 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.629 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.629 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.4s5 38/32 ECSubWrite(tid=21, reqid=client.4751.0:190, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2020-03-09T00:24:53.629 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.827011+0000", 2020-03-09T00:24:53.629 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7232159380000001, 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033358530000000002, 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827011+0000", 2020-03-09T00:24:53.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827011+0000", 2020-03-09T00:24:53.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1909999999999995e-05 2020-03-09T00:24:53.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827073+0000", 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2809000000000001e-05 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827085+0000", 2020-03-09T00:24:53.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9357999999999999e-05 2020-03-09T00:24:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827115+0000", 2020-03-09T00:24:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8720000000000002e-06 2020-03-09T00:24:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827123+0000", 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1547e-05 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827174+0000", 2020-03-09T00:24:53.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5359999999999992e-06 2020-03-09T00:24:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827183+0000", 2020-03-09T00:24:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031150050000000001 2020-03-09T00:24:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830298+0000", 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8816000000000002e-05 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830347+0000", 2020-03-09T00:24:53.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033358530000000002 2020-03-09T00:24:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s5 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:53.637 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.827839+0000", 2020-03-09T00:24:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7223873789999997, 2020-03-09T00:24:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030543599999999999, 2020-03-09T00:24:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827839+0000", 2020-03-09T00:24:53.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827839+0000", 2020-03-09T00:24:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113105 2020-03-09T00:24:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827952+0000", 2020-03-09T00:24:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3563000000000001e-05 2020-03-09T00:24:53.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827976+0000", 2020-03-09T00:24:53.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7655999999999998e-05 2020-03-09T00:24:53.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827994+0000", 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7170000000000008e-06 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828001+0000", 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5296e-05 2020-03-09T00:24:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828057+0000", 2020-03-09T00:24:53.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1810999999999999e-05 2020-03-09T00:24:53.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828068+0000", 2020-03-09T00:24:53.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027727699999999999 2020-03-09T00:24:53.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830841+0000", 2020-03-09T00:24:53.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2441999999999997e-05 2020-03-09T00:24:53.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830894+0000", 2020-03-09T00:24:53.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030543599999999999 2020-03-09T00:24:53.645 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s5 38/32 ECSubWrite(tid=11, reqid=client.4751.0:197, at_version=38'7, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.833173+0000", 2020-03-09T00:24:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.717053452, 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053212110000000002, 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.833173+0000", 2020-03-09T00:24:53.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.833173+0000", 2020-03-09T00:24:53.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5847000000000007e-05 2020-03-09T00:24:53.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.833259+0000", 2020-03-09T00:24:53.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1656999999999999e-05 2020-03-09T00:24:53.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.833281+0000", 2020-03-09T00:24:53.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6141e-05 2020-03-09T00:24:53.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.833297+0000", 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7179999999999999e-06 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.833304+0000", 2020-03-09T00:24:53.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0020776530000000001 2020-03-09T00:24:53.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835381+0000", 2020-03-09T00:24:53.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5855999999999999e-05 2020-03-09T00:24:53.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835397+0000", 2020-03-09T00:24:53.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030629989999999998 2020-03-09T00:24:53.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838460+0000", 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.434e-05 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838494+0000", 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053212110000000002 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.654 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s5 38/32 ECSubWrite(tid=12, reqid=client.4751.0:198, at_version=38'8, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.836540+0000", 2020-03-09T00:24:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7136864200000002, 2020-03-09T00:24:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001974824, 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836540+0000", 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836540+0000", 2020-03-09T00:24:53.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8134000000000003e-05 2020-03-09T00:24:53.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836598+0000", 2020-03-09T00:24:53.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9774000000000001e-05 2020-03-09T00:24:53.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836618+0000", 2020-03-09T00:24:53.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6186000000000001e-05 2020-03-09T00:24:53.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836634+0000", 2020-03-09T00:24:53.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0220000000000001e-06 2020-03-09T00:24:53.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836640+0000", 2020-03-09T00:24:53.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00080399900000000001 2020-03-09T00:24:53.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.837444+0000", 2020-03-09T00:24:53.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6340000000000001e-06 2020-03-09T00:24:53.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.837452+0000", 2020-03-09T00:24:53.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001045086 2020-03-09T00:24:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838497+0000", 2020-03-09T00:24:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7989e-05 2020-03-09T00:24:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838515+0000", 2020-03-09T00:24:53.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001974824 2020-03-09T00:24:53.663 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.663 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.663 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.664 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as6 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:24:53.664 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.864276+0000", 2020-03-09T00:24:53.664 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.6859505400000003, 2020-03-09T00:24:53.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053195380000000004, 2020-03-09T00:24:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864276+0000", 2020-03-09T00:24:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864276+0000", 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016136899999999999 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864438+0000", 2020-03-09T00:24:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6290000000000001e-05 2020-03-09T00:24:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864464+0000", 2020-03-09T00:24:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8046999999999998e-05 2020-03-09T00:24:53.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864482+0000", 2020-03-09T00:24:53.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3189999999999995e-06 2020-03-09T00:24:53.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864490+0000", 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1222000000000004e-05 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864551+0000", 2020-03-09T00:24:53.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.274e-05 2020-03-09T00:24:53.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864564+0000", 2020-03-09T00:24:53.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049779669999999998 2020-03-09T00:24:53.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869542+0000", 2020-03-09T00:24:53.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3584000000000001e-05 2020-03-09T00:24:53.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869596+0000", 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053195380000000004 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.673 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.10.23:2 3.fs0 3:f1702e73:::smithi10428818-39:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e39)", 2020-03-09T00:24:53.673 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:48.821062+0000", 2020-03-09T00:24:53.673 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7291645180000002, 2020-03-09T00:24:53.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045815029999999998, 2020-03-09T00:24:53.673 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.673 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:24:53.673 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821062+0000", 2020-03-09T00:24:53.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821062+0000", 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9005999999999997e-05 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821121+0000", 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4640000000000003e-06 2020-03-09T00:24:53.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821128+0000", 2020-03-09T00:24:53.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1800999999999999e-05 2020-03-09T00:24:53.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821139+0000", 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3749999999999997e-06 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821147+0000", 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1252e-05 2020-03-09T00:24:53.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821238+0000", 2020-03-09T00:24:53.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3989999999999993e-06 2020-03-09T00:24:53.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:24:53.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.821247+0000", 2020-03-09T00:24:53.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021526090000000002 2020-03-09T00:24:53.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.823400+0000", 2020-03-09T00:24:53.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000129701 2020-03-09T00:24:53.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.823530+0000", 2020-03-09T00:24:53.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021138960000000001 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.825644+0000", 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045815029999999998 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.682 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.7s6 42/32 ECSubWrite(tid=33, reqid=client.4751.0:224, at_version=0'0, trim_to=0'0, roll_forward_to=42'6))", 2020-03-09T00:24:53.682 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.835289+0000", 2020-03-09T00:24:53.682 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.714938106, 2020-03-09T00:24:53.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003132653, 2020-03-09T00:24:53.682 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.683 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.683 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835289+0000", 2020-03-09T00:24:53.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835289+0000", 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7256000000000002e-05 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835346+0000", 2020-03-09T00:24:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2682e-05 2020-03-09T00:24:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835359+0000", 2020-03-09T00:24:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5891000000000001e-05 2020-03-09T00:24:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835374+0000", 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6410000000000002e-06 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835382+0000", 2020-03-09T00:24:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9326000000000003e-05 2020-03-09T00:24:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835421+0000", 2020-03-09T00:24:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1092e-05 2020-03-09T00:24:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.835432+0000", 2020-03-09T00:24:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002957126 2020-03-09T00:24:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.838390+0000", 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1639e-05 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.838421+0000", 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003132653 2020-03-09T00:24:53.689 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.690 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.0s1 42/32 ECSubRead(tid=51, to_read={3:054f30c0:::smithi10428818-49:head=0,204800,0}, subchunks={3:054f30c0:::smithi10428818-49:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.851222+0000", 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.699005002, 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081548650000000007, 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851222+0000", 2020-03-09T00:24:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851222+0000", 2020-03-09T00:24:53.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5085000000000004e-05 2020-03-09T00:24:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851297+0000", 2020-03-09T00:24:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4131000000000002e-05 2020-03-09T00:24:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851331+0000", 2020-03-09T00:24:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5287e-05 2020-03-09T00:24:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851356+0000", 2020-03-09T00:24:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2993000000000002e-05 2020-03-09T00:24:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851379+0000", 2020-03-09T00:24:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000167641 2020-03-09T00:24:53.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851547+0000", 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0078297279999999993 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859377+0000", 2020-03-09T00:24:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081548650000000007 2020-03-09T00:24:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.697 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.0s1 42/32 ECSubRead(tid=53, to_read={3:054f30c0:::smithi10428818-49:head=0,204800,0}, subchunks={3:054f30c0:::smithi10428818-49:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.851961+0000", 2020-03-09T00:24:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.6982657729999999, 2020-03-09T00:24:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0076604539999999997, 2020-03-09T00:24:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851961+0000", 2020-03-09T00:24:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851961+0000", 2020-03-09T00:24:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7562000000000002e-05 2020-03-09T00:24:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852018+0000", 2020-03-09T00:24:53.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3115999999999999e-05 2020-03-09T00:24:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852032+0000", 2020-03-09T00:24:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9930999999999999e-05 2020-03-09T00:24:53.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852052+0000", 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4819999999999997e-06 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852059+0000", 2020-03-09T00:24:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073552899999999996 2020-03-09T00:24:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859414+0000", 2020-03-09T00:24:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000207073 2020-03-09T00:24:53.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859621+0000", 2020-03-09T00:24:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0076604539999999997 2020-03-09T00:24:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.704 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.0s1 42/32 ECSubRead(tid=54, to_read={3:054f30c0:::smithi10428818-49:head=0,204800,0}, subchunks={3:054f30c0:::smithi10428818-49:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:24:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.852554+0000", 2020-03-09T00:24:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.6976727089999999, 2020-03-09T00:24:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072931999999999997, 2020-03-09T00:24:53.705 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852554+0000", 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852554+0000", 2020-03-09T00:24:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3246999999999999e-05 2020-03-09T00:24:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852617+0000", 2020-03-09T00:24:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0971999999999999e-05 2020-03-09T00:24:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852628+0000", 2020-03-09T00:24:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0573000000000001e-05 2020-03-09T00:24:53.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852649+0000", 2020-03-09T00:24:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4439999999999996e-06 2020-03-09T00:24:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852655+0000", 2020-03-09T00:24:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0070002980000000003 2020-03-09T00:24:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859655+0000", 2020-03-09T00:24:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000191666 2020-03-09T00:24:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859847+0000", 2020-03-09T00:24:53.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072931999999999997 2020-03-09T00:24:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s5 42/32 ECSubWrite(tid=21, reqid=osd.15.0:21, at_version=42'14, trim_to=0'0, roll_forward_to=40'12))", 2020-03-09T00:24:53.713 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.853288+0000", 2020-03-09T00:24:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.69693866, 2020-03-09T00:24:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030041379999999999, 2020-03-09T00:24:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853288+0000", 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853288+0000", 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5141000000000004e-05 2020-03-09T00:24:53.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853363+0000", 2020-03-09T00:24:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.7603e-05 2020-03-09T00:24:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853391+0000", 2020-03-09T00:24:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7788e-05 2020-03-09T00:24:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853409+0000", 2020-03-09T00:24:53.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4069999999999993e-06 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853417+0000", 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0472999999999997e-05 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853457+0000", 2020-03-09T00:24:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9510000000000006e-06 2020-03-09T00:24:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.853465+0000", 2020-03-09T00:24:53.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0027709150000000001 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.856236+0000", 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5859999999999997e-05 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.856292+0000", 2020-03-09T00:24:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030041379999999999 2020-03-09T00:24:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s6 42/32 ECSubWrite(tid=42, reqid=osd.13.0:42, at_version=42'9, trim_to=0'0, roll_forward_to=41'7))", 2020-03-09T00:24:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.855033+0000", 2020-03-09T00:24:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.695193715, 2020-03-09T00:24:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002345995, 2020-03-09T00:24:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.722 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855033+0000", 2020-03-09T00:24:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855033+0000", 2020-03-09T00:24:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0151e-05 2020-03-09T00:24:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855063+0000", 2020-03-09T00:24:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2523000000000001e-05 2020-03-09T00:24:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855086+0000", 2020-03-09T00:24:53.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0346000000000001e-05 2020-03-09T00:24:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855096+0000", 2020-03-09T00:24:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1099999999999996e-06 2020-03-09T00:24:53.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855100+0000", 2020-03-09T00:24:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0683999999999998e-05 2020-03-09T00:24:53.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855151+0000", 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5074e-05 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855166+0000", 2020-03-09T00:24:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021612609999999998 2020-03-09T00:24:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.857327+0000", 2020-03-09T00:24:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1845999999999998e-05 2020-03-09T00:24:53.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.857379+0000", 2020-03-09T00:24:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002345995 2020-03-09T00:24:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.730 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:236 3.fs0 3:f75fa7c5:::smithi10428818-15:head [delete] snapc 9=[9,7,5,4] ondisk+write+peerstat_old+known_if_redirected e43)", 2020-03-09T00:24:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:52.835420+0000", 2020-03-09T00:24:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.71480696300000002, 2020-03-09T00:24:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053494859999999997, 2020-03-09T00:24:53.731 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 236 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:53.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835420+0000", 2020-03-09T00:24:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:53.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835420+0000", 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9057e-05 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835449+0000", 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9280000000000004e-06 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835456+0000", 2020-03-09T00:24:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3156000000000001e-05 2020-03-09T00:24:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:53.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835469+0000", 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2439999999999993e-06 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835477+0000", 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5022999999999998e-05 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835532+0000", 2020-03-09T00:24:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103707 2020-03-09T00:24:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:53.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.835636+0000", 2020-03-09T00:24:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00090928799999999996 2020-03-09T00:24:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.836545+0000", 2020-03-09T00:24:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074212499999999995 2020-03-09T00:24:53.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.837287+0000", 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033997089999999999 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:24:53.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.840687+0000", 2020-03-09T00:24:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2249000000000005e-05 2020-03-09T00:24:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.840769+0000", 2020-03-09T00:24:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053494859999999997 2020-03-09T00:24:53.740 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:53.741 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:53.741 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:53.743 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:53.741+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:53.743 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:24:53.841 INFO:tasks.rados.rados.0.smithi104.stdout:92: done (1 left) 2020-03-09T00:24:53.841 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 20 current snap is 6 2020-03-09T00:24:53.842 INFO:tasks.rados.rados.0.smithi104.stdout:93: seq_num 60 ranges {327680=638976,966656=376832} 2020-03-09T00:24:53.844 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:53.842+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:53.846 INFO:tasks.rados.rados.0.smithi104.stdout:93: writing smithi10428818-20 from 327680 to 966656 tid 1 2020-03-09T00:24:53.849 INFO:tasks.rados.rados.0.smithi104.stdout:93: writing smithi10428818-20 from 966656 to 1343488 tid 2 2020-03-09T00:24:53.849 INFO:tasks.rados.rados.0.smithi104.stdout:94: rollback oid 39 current snap is 6 2020-03-09T00:24:53.849 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 39 to 2 2020-03-09T00:24:53.850 INFO:tasks.rados.rados.0.smithi104.stdout:91: finishing write tid 1 to smithi10428818-18 2020-03-09T00:24:53.850 INFO:tasks.rados.rados.0.smithi104.stdout:91: finishing write tid 2 to smithi10428818-18 2020-03-09T00:24:53.850 INFO:tasks.rados.rados.0.smithi104.stdout:91: finishing write tid 4 to smithi10428818-18 2020-03-09T00:24:53.850 INFO:tasks.rados.rados.0.smithi104.stdout:91: finishing write tid 5 to smithi10428818-18 2020-03-09T00:24:53.850 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 18 v 12 (ObjNum 59 snap 6 seq_num 59) dirty exists 2020-03-09T00:24:53.850 INFO:tasks.rados.rados.0.smithi104.stdout:91: left oid 18 (ObjNum 59 snap 6 seq_num 59) 2020-03-09T00:24:53.850 INFO:tasks.rados.rados.0.smithi104.stdout:91: done (2 left) 2020-03-09T00:24:53.851 INFO:tasks.rados.rados.0.smithi104.stdout:95: copy_from oid 46 from oid 26 current snap is 6 2020-03-09T00:24:53.851 INFO:tasks.rados.rados.0.smithi104.stdout:96: copy_from oid 44 from oid 43 current snap is 6 2020-03-09T00:24:53.851 INFO:tasks.rados.rados.0.smithi104.stdout:97: delete oid 23 current snap is 6 2020-03-09T00:24:53.860 INFO:tasks.rados.rados.0.smithi104.stdout:97: done (4 left) 2020-03-09T00:24:53.860 INFO:tasks.rados.rados.0.smithi104.stdout:98: setattr oid 48 current snap is 6 2020-03-09T00:24:53.861 INFO:tasks.rados.rados.0.smithi104.stdout:94: finishing rollback tid 0 to smithi10428818-39 2020-03-09T00:24:53.862 INFO:tasks.rados.rados.0.smithi104.stdout:94: finishing rollback tid 1 to smithi10428818-39 2020-03-09T00:24:53.862 INFO:tasks.rados.rados.0.smithi104.stdout:94: finishing rollback tid 2 to smithi10428818-39 2020-03-09T00:24:53.862 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 39 v 21 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-03-09T00:24:53.863 INFO:tasks.rados.rados.0.smithi104.stdout:94: done (4 left) 2020-03-09T00:24:53.863 INFO:tasks.rados.rados.0.smithi104.stdout:99: read oid 38 snap -1 2020-03-09T00:24:53.863 INFO:tasks.rados.rados.0.smithi104.stdout:99: expect (ObjNum 37 snap 0 seq_num 37) 2020-03-09T00:24:53.863 INFO:tasks.rados.rados.0.smithi104.stdout:100: read oid 24 snap 5 2020-03-09T00:24:53.863 INFO:tasks.rados.rados.0.smithi104.stdout:100: expect (ObjNum 23 snap 0 seq_num 23) 2020-03-09T00:24:53.863 INFO:tasks.rados.rados.0.smithi104.stdout:101: snap_create 2020-03-09T00:24:53.866 INFO:tasks.rados.rados.0.smithi104.stdout:93: finishing write tid 1 to smithi10428818-20 2020-03-09T00:24:53.869 INFO:tasks.rados.rados.0.smithi104.stdout:93: finishing write tid 2 to smithi10428818-20 2020-03-09T00:24:53.870 INFO:tasks.rados.rados.0.smithi104.stdout:93: finishing write tid 4 to smithi10428818-20 2020-03-09T00:24:53.872 INFO:tasks.rados.rados.0.smithi104.stdout:93: finishing write tid 5 to smithi10428818-20 2020-03-09T00:24:53.872 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 20 v 18 (ObjNum 60 snap 6 seq_num 60) dirty exists 2020-03-09T00:24:53.872 INFO:tasks.rados.rados.0.smithi104.stdout:93: left oid 20 (ObjNum 60 snap 6 seq_num 60) 2020-03-09T00:24:53.873 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 48 v 23 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-03-09T00:24:53.873 INFO:tasks.rados.rados.0.smithi104.stdout:100: expect (ObjNum 23 snap 0 seq_num 23) 2020-03-09T00:24:53.884 INFO:tasks.rados.rados.0.smithi104.stdout:96: finishing copy_from to smithi10428818-44 2020-03-09T00:24:53.884 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 44 v 5 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-03-09T00:24:53.884 INFO:tasks.rados.rados.0.smithi104.stdout:95: finishing copy_from to smithi10428818-46 2020-03-09T00:24:53.884 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 46 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-03-09T00:24:53.884 INFO:tasks.rados.rados.0.smithi104.stdout:96: finishing copy_from racing read to smithi10428818-44 2020-03-09T00:24:53.884 INFO:tasks.rados.rados.0.smithi104.stdout:95: finishing copy_from racing read to smithi10428818-46 2020-03-09T00:24:53.884 INFO:tasks.rados.rados.0.smithi104.stdout:99: expect (ObjNum 37 snap 0 seq_num 37) 2020-03-09T00:24:53.888 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:53.888 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:53.888 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:24:53.888 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:53.897 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:24:53.939 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:24:53.940 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-03-09T00:24:53.945 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:53.944+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:54.013 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:24:54.013 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:24:54.014 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:54.021 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:24:54.046 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:54.045+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:54.082 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:54.104 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:54.124 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:54.144 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:54.148 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:24:54.146+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:24:54.164 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:54.169 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:24:54.170 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:24:54.170 INFO:tasks.thrashosds.thrasher:Removing osd 13, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2020-03-09T00:24:54.170 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd out 13 2020-03-09T00:24:54.180 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:54.180 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:24:54.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:24:54.180 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:24:54.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.181 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:190 3.4s0 3:231af75d:::smithi10428818-36:head [copy-from ver 4] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:54.181 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.811600+0000", 2020-03-09T00:24:54.181 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.3660699279999999, 2020-03-09T00:24:54.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020099, 2020-03-09T00:24:54.181 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.181 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:54.182 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.182 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.182 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.182 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 190 2020-03-09T00:24:54.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.182 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811600+0000", 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811600+0000", 2020-03-09T00:24:54.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.5548000000000001e-05 2020-03-09T00:24:54.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811695+0000", 2020-03-09T00:24:54.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2072999999999999e-05 2020-03-09T00:24:54.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811708+0000", 2020-03-09T00:24:54.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1101e-05 2020-03-09T00:24:54.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811729+0000", 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0558e-05 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811739+0000", 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7218999999999994e-05 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811816+0000", 2020-03-09T00:24:54.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.167e-05 2020-03-09T00:24:54.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2020-03-09T00:24:54.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811828+0000", 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001632494 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.813461+0000", 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3468999999999997e-05 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.813544+0000", 2020-03-09T00:24:54.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095589279999999995 2020-03-09T00:24:54.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.823103+0000", 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023422830000000001 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.825445+0000", 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010797929999999999 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826525+0000", 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4963000000000005e-05 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.826620+0000", 2020-03-09T00:24:54.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020099 2020-03-09T00:24:54.192 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.192 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.192 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.192 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:191 3.4s0 3:231af75d:::smithi10428818-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:24:54.192 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.811808+0000", 2020-03-09T00:24:54.193 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.3658615100000002, 2020-03-09T00:24:54.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016080786, 2020-03-09T00:24:54.193 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.193 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:54.193 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.193 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.193 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.194 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 191 2020-03-09T00:24:54.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.194 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811808+0000", 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811808+0000", 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5652000000000001e-05 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811834+0000", 2020-03-09T00:24:54.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2379999999999999e-06 2020-03-09T00:24:54.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811838+0000", 2020-03-09T00:24:54.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0790000000000001e-06 2020-03-09T00:24:54.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811846+0000", 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5300000000000001e-06 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811850+0000", 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9886999999999998e-05 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811910+0000", 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3320000000000003e-06 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.811917+0000", 2020-03-09T00:24:54.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017839850000000001 2020-03-09T00:24:54.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.813701+0000", 2020-03-09T00:24:54.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8195999999999996e-05 2020-03-09T00:24:54.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.813769+0000", 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011410643999999999 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.825180+0000", 2020-03-09T00:24:54.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012587899999999999 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.825306+0000", 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022718460000000001 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827578+0000", 2020-03-09T00:24:54.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000117228 2020-03-09T00:24:54.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827695+0000", 2020-03-09T00:24:54.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019429000000000001 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827889+0000", 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016080786 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.837845+0000", 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.3398249760000001, 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068359600000000003, 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.205 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:54.206 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837845+0000", 2020-03-09T00:24:54.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837845+0000", 2020-03-09T00:24:54.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013475299999999999 2020-03-09T00:24:54.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837980+0000", 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1409999999999999e-05 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.838011+0000", 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8563000000000002e-05 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.838030+0000", 2020-03-09T00:24:54.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3599999999999996e-06 2020-03-09T00:24:54.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.838038+0000", 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4095000000000002e-05 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.838102+0000", 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4208e-05 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.838116+0000", 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0065136700000000001 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844630+0000", 2020-03-09T00:24:54.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0900999999999999e-05 2020-03-09T00:24:54.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844681+0000", 2020-03-09T00:24:54.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068359600000000003 2020-03-09T00:24:54.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.212 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.213 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.213 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:210 3.bs0 3:d0a81a8b:::smithi10428818-46:head [copy-from ver 6] snapc 5=[5,4] ondisk+write+known_if_redirected e39)", 2020-03-09T00:24:54.213 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:48.817226+0000", 2020-03-09T00:24:54.213 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3604435940000004, 2020-03-09T00:24:54.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016693712999999999, 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 210 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.214 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817226+0000", 2020-03-09T00:24:54.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817226+0000", 2020-03-09T00:24:54.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0787000000000001e-05 2020-03-09T00:24:54.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817267+0000", 2020-03-09T00:24:54.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1510000000000001e-06 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817273+0000", 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2736e-05 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817286+0000", 2020-03-09T00:24:54.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8650000000000001e-06 2020-03-09T00:24:54.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817294+0000", 2020-03-09T00:24:54.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2994e-05 2020-03-09T00:24:54.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817357+0000", 2020-03-09T00:24:54.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.413e-06 2020-03-09T00:24:54.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817364+0000", 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0025948350000000002 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.819959+0000", 2020-03-09T00:24:54.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000144561 2020-03-09T00:24:54.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.820104+0000", 2020-03-09T00:24:54.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0082615339999999992 2020-03-09T00:24:54.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.828365+0000", 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048814210000000004 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.833247+0000", 2020-03-09T00:24:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00056600900000000002 2020-03-09T00:24:54.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:24:54.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.833813+0000", 2020-03-09T00:24:54.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000107407 2020-03-09T00:24:54.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.833920+0000", 2020-03-09T00:24:54.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016693712999999999 2020-03-09T00:24:54.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.224 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:211 3.bs0 3:d0a81a8b:::smithi10428818-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e39)", 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:48.817346+0000", 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3603238959999997, 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017632238000000001, 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.225 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 211 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817346+0000", 2020-03-09T00:24:54.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817346+0000", 2020-03-09T00:24:54.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2685000000000003e-05 2020-03-09T00:24:54.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817379+0000", 2020-03-09T00:24:54.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4283e-05 2020-03-09T00:24:54.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817393+0000", 2020-03-09T00:24:54.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2339e-05 2020-03-09T00:24:54.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817405+0000", 2020-03-09T00:24:54.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6580000000000001e-06 2020-03-09T00:24:54.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817410+0000", 2020-03-09T00:24:54.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2867000000000001e-05 2020-03-09T00:24:54.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817463+0000", 2020-03-09T00:24:54.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0222e-05 2020-03-09T00:24:54.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.817473+0000", 2020-03-09T00:24:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0029010609999999999 2020-03-09T00:24:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.820374+0000", 2020-03-09T00:24:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9338000000000001e-05 2020-03-09T00:24:54.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2020-03-09T00:24:54.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.820423+0000", 2020-03-09T00:24:54.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012606894 2020-03-09T00:24:54.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.833030+0000", 2020-03-09T00:24:54.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012313600000000001 2020-03-09T00:24:54.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:24:54.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.833153+0000", 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0016012299999999999 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.834755+0000", 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000105554 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.236 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.236 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.834860+0000", 2020-03-09T00:24:54.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000117971 2020-03-09T00:24:54.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.236 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.236 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:48.834978+0000", 2020-03-09T00:24:54.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017632238000000001 2020-03-09T00:24:54.237 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.237 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.237 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.237 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:229 3.4s0 3:231af75d:::smithi10428818-36:7 [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:51.849114+0000", 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.3285557400000001, 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0078719189999999998, 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.238 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 229 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849114+0000", 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849114+0000", 2020-03-09T00:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9400000000000002e-05 2020-03-09T00:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849154+0000", 2020-03-09T00:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0549999999999999e-06 2020-03-09T00:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849159+0000", 2020-03-09T00:24:54.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9039999999999998e-06 2020-03-09T00:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849167+0000", 2020-03-09T00:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.148e-06 2020-03-09T00:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849175+0000", 2020-03-09T00:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3771999999999999e-05 2020-03-09T00:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849228+0000", 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0178e-05 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "op must wait for map", 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849239+0000", 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002349823 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.851588+0000", 2020-03-09T00:24:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017029600000000001 2020-03-09T00:24:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.851759+0000", 2020-03-09T00:24:54.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0052273429999999997 2020-03-09T00:24:54.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.856986+0000", 2020-03-09T00:24:54.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0078719189999999998 2020-03-09T00:24:54.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.247 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.247 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:230 3.4s0 3:231af75d:::smithi10428818-36:7 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:24:54.247 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:51.849210+0000", 2020-03-09T00:24:54.247 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.3284598380000001, 2020-03-09T00:24:54.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0082499619999999996, 2020-03-09T00:24:54.248 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.248 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:54.248 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.248 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.248 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 230 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849210+0000", 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849210+0000", 2020-03-09T00:24:54.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7286000000000005e-05 2020-03-09T00:24:54.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849287+0000", 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8629999999999997e-06 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849296+0000", 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5104e-05 2020-03-09T00:24:54.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849311+0000", 2020-03-09T00:24:54.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7599999999999997e-06 2020-03-09T00:24:54.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849318+0000", 2020-03-09T00:24:54.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6279999999999997e-05 2020-03-09T00:24:54.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849364+0000", 2020-03-09T00:24:54.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7999999999999993e-06 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849374+0000", 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003087617 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852462+0000", 2020-03-09T00:24:54.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011502000000000001 2020-03-09T00:24:54.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.255 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852577+0000", 2020-03-09T00:24:54.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048832320000000004 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.857460+0000", 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0082499619999999996 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.257 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.257 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:231 3.4s0 3:231af75d:::smithi10428818-36:7 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:24:54.257 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:51.849373+0000", 2020-03-09T00:24:54.257 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.3282963799999998, 2020-03-09T00:24:54.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0091343870000000008, 2020-03-09T00:24:54.257 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 231 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849373+0000", 2020-03-09T00:24:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849373+0000", 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2890000000000001e-05 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849456+0000", 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8630000000000008e-06 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849464+0000", 2020-03-09T00:24:54.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3906e-05 2020-03-09T00:24:54.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849478+0000", 2020-03-09T00:24:54.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1290000000000001e-06 2020-03-09T00:24:54.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849484+0000", 2020-03-09T00:24:54.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5382000000000001e-05 2020-03-09T00:24:54.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849520+0000", 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8600000000000004e-06 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:24:54.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849527+0000", 2020-03-09T00:24:54.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0038638700000000002 2020-03-09T00:24:54.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.853390+0000", 2020-03-09T00:24:54.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000118897 2020-03-09T00:24:54.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.853509+0000", 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0049985899999999998 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.858508+0000", 2020-03-09T00:24:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0091343870000000008 2020-03-09T00:24:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s5 42/32 ECSubWrite(tid=52, reqid=client.4751.0:228, at_version=42'17, trim_to=0'0, roll_forward_to=38'16))", 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:51.852277+0000", 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.3253925369999999, 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003561453, 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.267 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852277+0000", 2020-03-09T00:24:54.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.268 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852277+0000", 2020-03-09T00:24:54.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6860000000000006e-05 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852354+0000", 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7552000000000001e-05 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852372+0000", 2020-03-09T00:24:54.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7140000000000005e-06 2020-03-09T00:24:54.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852381+0000", 2020-03-09T00:24:54.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4139999999999996e-06 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852386+0000", 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9925000000000001e-05 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.271 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.272 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852426+0000", 2020-03-09T00:24:54.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3022e-05 2020-03-09T00:24:54.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.272 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.272 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852439+0000", 2020-03-09T00:24:54.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033523239999999998 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.855791+0000", 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7642000000000002e-05 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.274 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.855839+0000", 2020-03-09T00:24:54.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003561453 2020-03-09T00:24:54.274 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.274 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.274 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.275 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:237 3.bs0 3:d89be7a5:::smithi10428818-18:head [append 0~622592] snapc 9=[9,7,5,4] ondisk+write+known_if_redirected e43)", 2020-03-09T00:24:54.275 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.846406+0000", 2020-03-09T00:24:54.275 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3312636250000001, 2020-03-09T00:24:54.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011368909, 2020-03-09T00:24:54.275 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.275 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:54.275 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 237 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.276 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.846406+0000", 2020-03-09T00:24:54.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.277 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.846406+0000", 2020-03-09T00:24:54.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00093874200000000001 2020-03-09T00:24:54.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.278 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.847345+0000", 2020-03-09T00:24:54.278 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6179999999999995e-06 2020-03-09T00:24:54.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.847354+0000", 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8862999999999999e-05 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.847372+0000", 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0439999999999996e-06 2020-03-09T00:24:54.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.280 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.280 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.847381+0000", 2020-03-09T00:24:54.280 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6999000000000001e-05 2020-03-09T00:24:54.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.280 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.847438+0000", 2020-03-09T00:24:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6873000000000006e-05 2020-03-09T00:24:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.847534+0000", 2020-03-09T00:24:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014714940000000001 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849006+0000", 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0079840650000000003 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.282 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.283 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856990+0000", 2020-03-09T00:24:54.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00068152800000000004 2020-03-09T00:24:54.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.283 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:24:54.283 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857671+0000", 2020-03-09T00:24:54.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000103683 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857775+0000", 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011368909 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.284 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.285 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:238 3.bs0 3:d89be7a5:::smithi10428818-18:head [append 0~376832] snapc 9=[9,7,5,4] ondisk+write+known_if_redirected e43)", 2020-03-09T00:24:54.285 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.848312+0000", 2020-03-09T00:24:54.285 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3293574530000001, 2020-03-09T00:24:54.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010454216000000001, 2020-03-09T00:24:54.285 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.286 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:54.286 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.286 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.286 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.286 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 238 2020-03-09T00:24:54.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.286 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848312+0000", 2020-03-09T00:24:54.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848312+0000", 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051768300000000001 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848830+0000", 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3340000000000002e-06 2020-03-09T00:24:54.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848838+0000", 2020-03-09T00:24:54.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1866999999999998e-05 2020-03-09T00:24:54.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848870+0000", 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1740000000000003e-06 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848877+0000", 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003757804 2020-03-09T00:24:54.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.291 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.291 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.852635+0000", 2020-03-09T00:24:54.291 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000130963 2020-03-09T00:24:54.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.291 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.852766+0000", 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010559079999999999 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853822+0000", 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047673209999999997 2020-03-09T00:24:54.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858589+0000", 2020-03-09T00:24:54.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000102755 2020-03-09T00:24:54.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:24:54.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858692+0000", 2020-03-09T00:24:54.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4406999999999994e-05 2020-03-09T00:24:54.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.294 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.295 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858767+0000", 2020-03-09T00:24:54.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010454216000000001 2020-03-09T00:24:54.295 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.295 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.295 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:239 3.bs0 3:d89be7a5:::smithi10428818-18:head [setxattr _header (56) in=7b] snapc 9=[9,7,5,4] ondisk+write+known_if_redirected e43)", 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.848961+0000", 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.328709097, 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.009992061, 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.296 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.297 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.297 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 239 2020-03-09T00:24:54.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.297 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848961+0000", 2020-03-09T00:24:54.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.848961+0000", 2020-03-09T00:24:54.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5659000000000001e-05 2020-03-09T00:24:54.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849046+0000", 2020-03-09T00:24:54.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0750000000000004e-06 2020-03-09T00:24:54.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849055+0000", 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4514e-05 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849070+0000", 2020-03-09T00:24:54.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4480000000000002e-06 2020-03-09T00:24:54.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849075+0000", 2020-03-09T00:24:54.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0069293990000000001 2020-03-09T00:24:54.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.302 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.302 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856005+0000", 2020-03-09T00:24:54.302 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000129701 2020-03-09T00:24:54.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.302 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.302 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856135+0000", 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00054722400000000002 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856682+0000", 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020889250000000002 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858771+0000", 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000117547 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858888+0000", 2020-03-09T00:24:54.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4568999999999998e-05 2020-03-09T00:24:54.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.305 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858953+0000", 2020-03-09T00:24:54.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.009992061 2020-03-09T00:24:54.305 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.305 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:240 3.bs0 3:d89be7a5:::smithi10428818-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e43)", 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.849132+0000", 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3285378510000001, 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011745762999999999, 2020-03-09T00:24:54.306 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.307 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:54.307 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.307 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.307 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.307 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 240 2020-03-09T00:24:54.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.307 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849132+0000", 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.308 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849132+0000", 2020-03-09T00:24:54.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5328999999999999e-05 2020-03-09T00:24:54.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.309 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849177+0000", 2020-03-09T00:24:54.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3159999999999998e-06 2020-03-09T00:24:54.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849182+0000", 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1871e-05 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849194+0000", 2020-03-09T00:24:54.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1780000000000002e-06 2020-03-09T00:24:54.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.849199+0000", 2020-03-09T00:24:54.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0089010040000000006 2020-03-09T00:24:54.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858100+0000", 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3002000000000002e-05 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:24:54.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.858193+0000", 2020-03-09T00:24:54.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001236006 2020-03-09T00:24:54.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.859429+0000", 2020-03-09T00:24:54.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010536500000000001 2020-03-09T00:24:54.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.314 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.859534+0000", 2020-03-09T00:24:54.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013436920000000001 2020-03-09T00:24:54.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.860878+0000", 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011745762999999999 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.315 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 43/32 ECSubWriteReply(tid=27, last_complete=43'10, committed=1, applied=1))", 2020-03-09T00:24:54.316 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.853087+0000", 2020-03-09T00:24:54.316 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3245827590000001, 2020-03-09T00:24:54.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003852411, 2020-03-09T00:24:54.316 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.316 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:54.316 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853087+0000", 2020-03-09T00:24:54.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853087+0000", 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9234999999999998e-05 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853146+0000", 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6989999999999998e-06 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853156+0000", 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8658999999999999e-05 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853175+0000", 2020-03-09T00:24:54.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4699999999999999e-06 2020-03-09T00:24:54.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.320 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.320 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853181+0000", 2020-03-09T00:24:54.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003727087 2020-03-09T00:24:54.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856908+0000", 2020-03-09T00:24:54.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1260999999999998e-05 2020-03-09T00:24:54.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856940+0000", 2020-03-09T00:24:54.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003852411 2020-03-09T00:24:54.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.322 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.322 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 43/32 ECSubWriteReply(tid=27, last_complete=43'10, committed=1, applied=1))", 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.853375+0000", 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3242945500000001, 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0036161539999999999, 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853375+0000", 2020-03-09T00:24:54.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853375+0000", 2020-03-09T00:24:54.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9609000000000001e-05 2020-03-09T00:24:54.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853405+0000", 2020-03-09T00:24:54.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7039999999999999e-06 2020-03-09T00:24:54.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853411+0000", 2020-03-09T00:24:54.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2829e-05 2020-03-09T00:24:54.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853423+0000", 2020-03-09T00:24:54.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7189999999999998e-06 2020-03-09T00:24:54.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.327 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853429+0000", 2020-03-09T00:24:54.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035442519999999999 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856973+0000", 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8040999999999999e-05 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.856991+0000", 2020-03-09T00:24:54.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0036161539999999999 2020-03-09T00:24:54.329 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.329 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.329 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.330 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 43/32 ECSubWriteReply(tid=27, last_complete=43'10, committed=1, applied=1))", 2020-03-09T00:24:54.330 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.853474+0000", 2020-03-09T00:24:54.330 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3241954069999999, 2020-03-09T00:24:54.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035549779999999999, 2020-03-09T00:24:54.330 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.330 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:54.330 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.331 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.331 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.331 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853474+0000", 2020-03-09T00:24:54.331 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.331 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.332 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853474+0000", 2020-03-09T00:24:54.332 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8466e-05 2020-03-09T00:24:54.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.332 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853523+0000", 2020-03-09T00:24:54.333 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8919999999999999e-06 2020-03-09T00:24:54.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.333 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.333 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853528+0000", 2020-03-09T00:24:54.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.576e-06 2020-03-09T00:24:54.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.334 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.334 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.334 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853535+0000", 2020-03-09T00:24:54.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6949999999999999e-06 2020-03-09T00:24:54.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853540+0000", 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0034772919999999999 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857017+0000", 2020-03-09T00:24:54.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2057e-05 2020-03-09T00:24:54.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.336 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.336 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857029+0000", 2020-03-09T00:24:54.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035549779999999999 2020-03-09T00:24:54.336 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.337 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.337 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.337 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.337 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 43/32 ECSubWriteReply(tid=27, last_complete=43'10, committed=1, applied=1))", 2020-03-09T00:24:54.338 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.853480+0000", 2020-03-09T00:24:54.338 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3241902480000001, 2020-03-09T00:24:54.338 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003589035, 2020-03-09T00:24:54.338 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.338 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:54.338 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853480+0000", 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853480+0000", 2020-03-09T00:24:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0547999999999999e-05 2020-03-09T00:24:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853520+0000", 2020-03-09T00:24:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1789999999999996e-06 2020-03-09T00:24:54.340 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853526+0000", 2020-03-09T00:24:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4619e-05 2020-03-09T00:24:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.341 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.342 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853541+0000", 2020-03-09T00:24:54.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3879999999999997e-06 2020-03-09T00:24:54.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.342 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.342 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853547+0000", 2020-03-09T00:24:54.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003509641 2020-03-09T00:24:54.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.343 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.343 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857057+0000", 2020-03-09T00:24:54.343 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.166e-05 2020-03-09T00:24:54.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857069+0000", 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003589035 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.344 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.345 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 43/32 ECSubWriteReply(tid=27, last_complete=43'10, committed=1, applied=1))", 2020-03-09T00:24:54.345 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.853626+0000", 2020-03-09T00:24:54.345 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3240439509999999, 2020-03-09T00:24:54.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037450040000000001, 2020-03-09T00:24:54.345 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.345 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:54.345 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.346 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853626+0000", 2020-03-09T00:24:54.346 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.347 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853626+0000", 2020-03-09T00:24:54.347 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9347999999999997e-05 2020-03-09T00:24:54.347 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.347 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.347 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853665+0000", 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7599999999999997e-06 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853672+0000", 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.206e-05 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.349 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.349 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853684+0000", 2020-03-09T00:24:54.349 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0100000000000003e-06 2020-03-09T00:24:54.349 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.349 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.349 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.349 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.853689+0000", 2020-03-09T00:24:54.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0036698630000000002 2020-03-09T00:24:54.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.350 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857359+0000", 2020-03-09T00:24:54.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1963e-05 2020-03-09T00:24:54.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.351 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.857371+0000", 2020-03-09T00:24:54.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037450040000000001 2020-03-09T00:24:54.351 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.351 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:249 3.bs0 3:d0a81a8b:::smithi10428818-46:head [copy-from ver 6] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.849133+0000", 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.32853641300000003, 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.026957286, 2020-03-09T00:24:54.352 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.353 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:54.353 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.353 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.353 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.353 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 249 2020-03-09T00:24:54.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.354 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.354 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.354 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849133+0000", 2020-03-09T00:24:54.354 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.354 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849133+0000", 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3349000000000005e-05 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849207+0000", 2020-03-09T00:24:54.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1499999999999999e-06 2020-03-09T00:24:54.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.356 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.356 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849215+0000", 2020-03-09T00:24:54.356 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0559e-05 2020-03-09T00:24:54.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849226+0000", 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2079999999999997e-06 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849232+0000", 2020-03-09T00:24:54.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00066397199999999998 2020-03-09T00:24:54.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.358 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.358 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849896+0000", 2020-03-09T00:24:54.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011097799999999999 2020-03-09T00:24:54.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850007+0000", 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017440982000000001 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867448+0000", 2020-03-09T00:24:54.359 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003338808 2020-03-09T00:24:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.870786+0000", 2020-03-09T00:24:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0051702969999999999 2020-03-09T00:24:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.875957+0000", 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013398300000000001 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.876091+0000", 2020-03-09T00:24:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.026957286 2020-03-09T00:24:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:250 3.bs0 3:d0a81a8b:::smithi10428818-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:24:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.849278+0000", 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.32839207100000001, 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.027836001999999999, 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:54.363 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 250 2020-03-09T00:24:54.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.364 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:54.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:54.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849278+0000", 2020-03-09T00:24:54.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:54.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849278+0000", 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4487000000000003e-05 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:54.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849322+0000", 2020-03-09T00:24:54.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1020000000000002e-06 2020-03-09T00:24:54.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:54.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849328+0000", 2020-03-09T00:24:54.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2648e-05 2020-03-09T00:24:54.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:54.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849341+0000", 2020-03-09T00:24:54.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8439999999999998e-06 2020-03-09T00:24:54.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.367 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.849349+0000", 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00089056400000000005 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850239+0000", 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9107000000000002e-05 2020-03-09T00:24:54.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2020-03-09T00:24:54.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850299+0000", 2020-03-09T00:24:54.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020340917999999999 2020-03-09T00:24:54.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.870639+0000", 2020-03-09T00:24:54.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013658400000000001 2020-03-09T00:24:54.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:24:54.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.870776+0000", 2020-03-09T00:24:54.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0061766829999999997 2020-03-09T00:24:54.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.371 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:54.371 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.876953+0000", 2020-03-09T00:24:54.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6616000000000001e-05 2020-03-09T00:24:54.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:24:54.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.877019+0000", 2020-03-09T00:24:54.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4449000000000003e-05 2020-03-09T00:24:54.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:54.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:54.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:54.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.877114+0000", 2020-03-09T00:24:54.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.027836001999999999 2020-03-09T00:24:54.373 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.373 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.373 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.374 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:54.374 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:54.374 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:54.376 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:54.376 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:54.376 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:54.376 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:54.377 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:54.377 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:54.377 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:54.377 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:54.377 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:54.330+0000 7f2999937700 0 -- 172.21.15.148:0/2878362561 >> [v2:172.21.15.50:6816/27631,v1:172.21.15.50:6817/27631] conn(0x7f297402c010 msgr2=0x7f297402e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:54.378 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:54.374+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:54.378 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:24:54.449 INFO:tasks.ceph.mon.b.smithi050.stderr:2020-03-09T00:24:54.445+0000 7fa09607d700 -1 mon.b@0(leader).osd e44 definitely_dead 0 2020-03-09T00:24:54.476 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:54.474+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:54.515 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:54.516 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:54.516 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:24:54.516 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:54.525 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:24:54.553 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:54.573 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:54.577 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:24:54.576+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:24:54.593 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:54.595 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.595 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.595 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.595 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.595 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.595 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.596 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.596 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.596 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.596 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.596 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.597 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.597 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.597 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.597 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.597 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:24:54.678 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:54.677+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:54.689 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:54.689 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:54.698 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:24:54.778 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:24:54.776+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:24:54.823 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:54.823 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:54.832 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:24:54.851 INFO:tasks.rados.rados.0.smithi104.stdout:93: done (6 left) 2020-03-09T00:24:54.851 INFO:tasks.rados.rados.0.smithi104.stdout:95: done (5 left) 2020-03-09T00:24:54.851 INFO:tasks.rados.rados.0.smithi104.stdout:96: done (4 left) 2020-03-09T00:24:54.851 INFO:tasks.rados.rados.0.smithi104.stdout:98: done (3 left) 2020-03-09T00:24:54.851 INFO:tasks.rados.rados.0.smithi104.stdout:99: done (2 left) 2020-03-09T00:24:54.851 INFO:tasks.rados.rados.0.smithi104.stdout:100: done (1 left) 2020-03-09T00:24:54.852 INFO:tasks.rados.rados.0.smithi104.stdout:101: done (0 left) 2020-03-09T00:24:54.852 INFO:tasks.rados.rados.0.smithi104.stdout:102: copy_from oid 34 from oid 29 current snap is 7 2020-03-09T00:24:54.852 INFO:tasks.rados.rados.0.smithi104.stdout:103: snap_create 2020-03-09T00:24:54.870 INFO:teuthology.orchestra.run.smithi148.stderr:marked out osd.13. 2020-03-09T00:24:54.870 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is set 2020-03-09T00:24:54.879 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:54.878+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:54.958 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:54.958 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:54.967 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:24:54.980 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:54.978+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:55.081 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:55.080+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:55.082 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:55.082 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:55.091 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:24:55.182 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:55.180+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:55.206 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:55.206 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:55.214 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:24:55.283 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:55.281+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:55.343 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:55.343 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:55.352 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:24:55.383 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:55.382+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:55.467 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:55.467 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:55.475 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:24:55.484 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:55.483+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:55.585 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:55.582+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:55.590 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:55.590 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:55.599 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:24:55.687 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:55.685+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:55.726 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:55.726 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:55.734 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:24:55.787 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:24:55.786+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:24:55.850 INFO:tasks.rados.rados.0.smithi104.stdout:103: done (1 left) 2020-03-09T00:24:55.851 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 19 current snap is 8 2020-03-09T00:24:55.851 INFO:tasks.rados.rados.0.smithi104.stdout:104: seq_num 62 ranges {311296=671744,983040=376832} 2020-03-09T00:24:55.858 INFO:tasks.rados.rados.0.smithi104.stdout:104: writing smithi10428818-19 from 311296 to 983040 tid 1 2020-03-09T00:24:55.861 INFO:tasks.rados.rados.0.smithi104.stdout:104: writing smithi10428818-19 from 983040 to 1359872 tid 2 2020-03-09T00:24:55.861 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 12 current snap is 8 2020-03-09T00:24:55.861 INFO:tasks.rados.rados.0.smithi104.stdout:105: seq_num 63 ranges {196608=688128,884736=376832} 2020-03-09T00:24:55.866 INFO:tasks.rados.rados.0.smithi104.stdout:105: writing smithi10428818-12 from 196608 to 884736 tid 1 2020-03-09T00:24:55.868 INFO:tasks.rados.rados.0.smithi104.stdout:105: writing smithi10428818-12 from 884736 to 1261568 tid 2 2020-03-09T00:24:55.868 INFO:tasks.rados.rados.0.smithi104.stdout:106: snap_remove snap 6 2020-03-09T00:24:55.888 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:55.887+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:55.896 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:55.896 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:55.906 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:24:55.989 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:55.988+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:56.062 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:56.062 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:56.070 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:24:56.089 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:56.088+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:56.191 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:56.189+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:56.208 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:56.209 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:56.218 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:24:56.292 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:56.290+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:56.341 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:56.341 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:56.350 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:24:56.393 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:24:56.391+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:24:56.494 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:56.492+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:56.514 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:56.514 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:56.523 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:24:56.595 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:56.593+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:56.604 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:24:56.653 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:56.653 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:56.663 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:24:56.706 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:56.705+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:56.744 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:56.763 INFO:tasks.rados.rados.0.smithi104.stdout:106: done (3 left) 2020-03-09T00:24:56.763 INFO:tasks.rados.rados.0.smithi104.stdout:107: snap_remove snap 5 2020-03-09T00:24:56.772 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:56.796 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:24:56.793+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:24:56.803 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:56.804 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:56.804 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:24:56.804 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:24:56.804 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:56.813 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:56.815 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:24:56.833 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:56.859 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:56.871 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-03-09T00:24:56.881 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:56.897 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:56.896+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:56.905 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:56.925 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:56.945 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:56.966 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:56.966 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:24:56.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:24:56.967 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:24:56.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.967 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 38/32 ECSubWrite(tid=32, reqid=client.4751.0:187, at_version=38'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:56.967 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.818690+0000", 2020-03-09T00:24:56.967 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.1447209750000003, 2020-03-09T00:24:56.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028896970000000001, 2020-03-09T00:24:56.967 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818690+0000", 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818690+0000", 2020-03-09T00:24:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4416000000000001e-05 2020-03-09T00:24:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:56.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818754+0000", 2020-03-09T00:24:56.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5262999999999999e-05 2020-03-09T00:24:56.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:56.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818779+0000", 2020-03-09T00:24:56.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5274999999999999e-05 2020-03-09T00:24:56.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818794+0000", 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9199999999999998e-06 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:56.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818801+0000", 2020-03-09T00:24:56.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0927999999999998e-05 2020-03-09T00:24:56.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:56.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818852+0000", 2020-03-09T00:24:56.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2754e-05 2020-03-09T00:24:56.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:56.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818865+0000", 2020-03-09T00:24:56.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0026691520000000002 2020-03-09T00:24:56.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:56.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.821534+0000", 2020-03-09T00:24:56.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4988999999999999e-05 2020-03-09T00:24:56.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:56.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.821579+0000", 2020-03-09T00:24:56.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028896970000000001 2020-03-09T00:24:56.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:56.975 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:56.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:56.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.975 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 38/32 ECSubWrite(tid=33, reqid=client.4751.0:188, at_version=38'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:56.975 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.818856+0000", 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.1445547089999994, 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002768473, 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:56.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818856+0000", 2020-03-09T00:24:56.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:56.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:56.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818856+0000", 2020-03-09T00:24:56.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3944999999999999e-05 2020-03-09T00:24:56.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818910+0000", 2020-03-09T00:24:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8207999999999999e-05 2020-03-09T00:24:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818928+0000", 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4785e-05 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818943+0000", 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0559999999999999e-06 2020-03-09T00:24:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.818950+0000", 2020-03-09T00:24:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0016882430000000001 2020-03-09T00:24:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:56.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.820638+0000", 2020-03-09T00:24:56.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.153e-05 2020-03-09T00:24:56.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:56.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.820650+0000", 2020-03-09T00:24:56.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00093361900000000001 2020-03-09T00:24:56.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:56.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.821583+0000", 2020-03-09T00:24:56.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1087000000000002e-05 2020-03-09T00:24:56.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:56.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.821624+0000", 2020-03-09T00:24:56.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002768473 2020-03-09T00:24:56.987 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:56.988 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:56.988 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:56.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.988 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:56.988 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.837154+0000", 2020-03-09T00:24:56.988 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.1262565440000003, 2020-03-09T00:24:56.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045504289999999999, 2020-03-09T00:24:56.989 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:56.989 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:56.989 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:56.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:56.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837154+0000", 2020-03-09T00:24:56.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:56.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:56.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837154+0000", 2020-03-09T00:24:56.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000279912 2020-03-09T00:24:56.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:56.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837434+0000", 2020-03-09T00:24:56.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7867e-05 2020-03-09T00:24:56.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:56.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837452+0000", 2020-03-09T00:24:56.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0740999999999999e-05 2020-03-09T00:24:56.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:56.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837462+0000", 2020-03-09T00:24:56.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3679999999999998e-06 2020-03-09T00:24:56.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:56.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837469+0000", 2020-03-09T00:24:56.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7648e-05 2020-03-09T00:24:56.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:56.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837536+0000", 2020-03-09T00:24:56.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5393e-05 2020-03-09T00:24:56.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837552+0000", 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0041146919999999997 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.841667+0000", 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7808000000000001e-05 2020-03-09T00:24:56.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:56.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.841704+0000", 2020-03-09T00:24:56.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045504289999999999 2020-03-09T00:24:56.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:56.996 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:56.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:56.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.997 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 38/32 ECSubWrite(tid=16, reqid=client.4751.0:201, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:56.997 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.842955+0000", 2020-03-09T00:24:56.997 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.1204552999999997, 2020-03-09T00:24:56.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025537009999999998, 2020-03-09T00:24:56.998 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:56.998 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:56.998 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:56.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:56.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.842955+0000", 2020-03-09T00:24:56.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:56.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:56.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:56.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:56.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.842955+0000", 2020-03-09T00:24:56.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000105307 2020-03-09T00:24:56.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843060+0000", 2020-03-09T00:24:57.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8777e-05 2020-03-09T00:24:57.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843079+0000", 2020-03-09T00:24:57.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2435e-05 2020-03-09T00:24:57.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843092+0000", 2020-03-09T00:24:57.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0290000000000002e-06 2020-03-09T00:24:57.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843098+0000", 2020-03-09T00:24:57.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0070000000000001e-05 2020-03-09T00:24:57.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843138+0000", 2020-03-09T00:24:57.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1986e-05 2020-03-09T00:24:57.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843150+0000", 2020-03-09T00:24:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023084109999999998 2020-03-09T00:24:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.845458+0000", 2020-03-09T00:24:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0686000000000002e-05 2020-03-09T00:24:57.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.845509+0000", 2020-03-09T00:24:57.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025537009999999998 2020-03-09T00:24:57.005 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.005 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.006 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.006 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:57.006 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:24:57.006 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.1134499770000001, 2020-03-09T00:24:57.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999, 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017622699999999999 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850137+0000", 2020-03-09T00:24:57.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1974000000000003e-05 2020-03-09T00:24:57.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850169+0000", 2020-03-09T00:24:57.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4702000000000002e-05 2020-03-09T00:24:57.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850193+0000", 2020-03-09T00:24:57.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5790000000000004e-06 2020-03-09T00:24:57.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850202+0000", 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013641599999999999 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850338+0000", 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4691999999999999e-05 2020-03-09T00:24:57.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850353+0000", 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011849303 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862202+0000", 2020-03-09T00:24:57.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1013000000000001e-05 2020-03-09T00:24:57.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862253+0000", 2020-03-09T00:24:57.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999 2020-03-09T00:24:57.014 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.014 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.014 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.014 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:57.015 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:24:57.015 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.1072545730000001, 2020-03-09T00:24:57.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595, 2020-03-09T00:24:57.015 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.015 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.015 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:24:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:24:57.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107948 2020-03-09T00:24:57.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856264+0000", 2020-03-09T00:24:57.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4422000000000001e-05 2020-03-09T00:24:57.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856288+0000", 2020-03-09T00:24:57.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5265e-05 2020-03-09T00:24:57.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856304+0000", 2020-03-09T00:24:57.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3209999999999994e-06 2020-03-09T00:24:57.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856313+0000", 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00056587900000000003 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856879+0000", 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1976e-05 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856891+0000", 2020-03-09T00:24:57.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098355160000000007 2020-03-09T00:24:57.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866726+0000", 2020-03-09T00:24:57.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6267999999999996e-05 2020-03-09T00:24:57.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866802+0000", 2020-03-09T00:24:57.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595 2020-03-09T00:24:57.022 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=39, reqid=client.4751.0:244, at_version=44'18, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.863381+0000", 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.1000293210000001, 2020-03-09T00:24:57.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034871720000000002, 2020-03-09T00:24:57.024 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.024 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.024 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863381+0000", 2020-03-09T00:24:57.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863381+0000", 2020-03-09T00:24:57.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0792999999999999e-05 2020-03-09T00:24:57.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863442+0000", 2020-03-09T00:24:57.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4428000000000001e-05 2020-03-09T00:24:57.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863466+0000", 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7127999999999999e-05 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863484+0000", 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9419999999999998e-06 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863490+0000", 2020-03-09T00:24:57.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015948700000000001 2020-03-09T00:24:57.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863650+0000", 2020-03-09T00:24:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3923999999999999e-05 2020-03-09T00:24:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863664+0000", 2020-03-09T00:24:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003164053 2020-03-09T00:24:57.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866828+0000", 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0417000000000003e-05 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866868+0000", 2020-03-09T00:24:57.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034871720000000002 2020-03-09T00:24:57.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.031 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.031 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:57.032 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.864672+0000", 2020-03-09T00:24:57.032 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.0987387310000001, 2020-03-09T00:24:57.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0080159729999999992, 2020-03-09T00:24:57.032 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.032 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.032 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864672+0000", 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864672+0000", 2020-03-09T00:24:57.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018526400000000001 2020-03-09T00:24:57.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864857+0000", 2020-03-09T00:24:57.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8279000000000001e-05 2020-03-09T00:24:57.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864885+0000", 2020-03-09T00:24:57.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6443000000000002e-05 2020-03-09T00:24:57.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864902+0000", 2020-03-09T00:24:57.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6979999999999998e-06 2020-03-09T00:24:57.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864909+0000", 2020-03-09T00:24:57.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9886999999999998e-05 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864969+0000", 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4875000000000001e-05 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864984+0000", 2020-03-09T00:24:57.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0076501599999999996 2020-03-09T00:24:57.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.872634+0000", 2020-03-09T00:24:57.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3366999999999999e-05 2020-03-09T00:24:57.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.872688+0000", 2020-03-09T00:24:57.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0080159729999999992 2020-03-09T00:24:57.039 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.039 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.040 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.040 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/46 ECSubWrite(tid=26, reqid=client.4751.0:265, at_version=47'10, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:57.040 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.782000+0000", 2020-03-09T00:24:57.040 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.181410183, 2020-03-09T00:24:57.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088401729999999998, 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782000+0000", 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782000+0000", 2020-03-09T00:24:57.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00075502399999999997 2020-03-09T00:24:57.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782755+0000", 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5735999999999999e-05 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782781+0000", 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6680999999999998e-05 2020-03-09T00:24:57.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782798+0000", 2020-03-09T00:24:57.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6900000000000003e-06 2020-03-09T00:24:57.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782804+0000", 2020-03-09T00:24:57.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00042437399999999999 2020-03-09T00:24:57.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783229+0000", 2020-03-09T00:24:57.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4287000000000001e-05 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783243+0000", 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066180969999999999 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.789861+0000", 2020-03-09T00:24:57.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00097928399999999997 2020-03-09T00:24:57.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790840+0000", 2020-03-09T00:24:57.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088401729999999998 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(3.6s1 47/46 [PushOp(3:6178963a:::smithi10428818-25:head, version: 35'6, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:6178963a:::smithi10428818-25:head@35'6, size: 409600, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:409600, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.783556+0000", 2020-03-09T00:24:57.048 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.179854972, 2020-03-09T00:24:57.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028115879999999998, 2020-03-09T00:24:57.049 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.049 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.049 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783556+0000", 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783556+0000", 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014708000000000001 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783703+0000", 2020-03-09T00:24:57.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7255999999999999e-05 2020-03-09T00:24:57.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783720+0000", 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6520999999999999e-05 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783746+0000", 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8640000000000008e-06 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783756+0000", 2020-03-09T00:24:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5714000000000001e-05 2020-03-09T00:24:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783822+0000", 2020-03-09T00:24:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025451530000000001 2020-03-09T00:24:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.786367+0000", 2020-03-09T00:24:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028115879999999998 2020-03-09T00:24:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.055 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.055 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.055 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPushReply(1.0 47/46 [PushReplyOp(1:15e1a584:::INTEL_SSDPEDMD400G4_CVFT6233005G400BGN:head)])", 2020-03-09T00:24:57.055 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.784284+0000", 2020-03-09T00:24:57.055 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.17912671599999999, 2020-03-09T00:24:57.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035330359999999998, 2020-03-09T00:24:57.056 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.056 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.056 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.784284+0000", 2020-03-09T00:24:57.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.784284+0000", 2020-03-09T00:24:57.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6529e-05 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.784340+0000", 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.242e-06 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.784349+0000", 2020-03-09T00:24:57.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.433e-05 2020-03-09T00:24:57.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.784363+0000", 2020-03-09T00:24:57.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.874e-06 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.784369+0000", 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031904120000000001 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787559+0000", 2020-03-09T00:24:57.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00025764899999999999 2020-03-09T00:24:57.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787817+0000", 2020-03-09T00:24:57.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035330359999999998 2020-03-09T00:24:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/46 ECSubWrite(tid=27, reqid=client.4751.0:266, at_version=47'11, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:57.062 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.786926+0000", 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.17648453, 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042076170000000003, 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.786926+0000", 2020-03-09T00:24:57.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.786926+0000", 2020-03-09T00:24:57.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00034550500000000002 2020-03-09T00:24:57.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787271+0000", 2020-03-09T00:24:57.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6393e-05 2020-03-09T00:24:57.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787298+0000", 2020-03-09T00:24:57.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0356e-05 2020-03-09T00:24:57.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787318+0000", 2020-03-09T00:24:57.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0700000000000007e-06 2020-03-09T00:24:57.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787326+0000", 2020-03-09T00:24:57.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010933399999999999 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787436+0000", 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6299000000000001e-05 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.787452+0000", 2020-03-09T00:24:57.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003635812 2020-03-09T00:24:57.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.791088+0000", 2020-03-09T00:24:57.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5847999999999999e-05 2020-03-09T00:24:57.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.791134+0000", 2020-03-09T00:24:57.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042076170000000003 2020-03-09T00:24:57.070 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.070 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(3.4s6 47/46 [PushOp(3:231af75d:::smithi10428818-36:4, version: 38'3, data_included: [0~147456], data_size: 147456, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:231af75d:::smithi10428818-36:4@38'3, size: 589824, copy_subset: [], clone_subset: {}, snapset: 4=[]:{4=[4]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:589824, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.790162+0000", 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.173248701, 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034322279999999998, 2020-03-09T00:24:57.071 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.072 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.072 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790162+0000", 2020-03-09T00:24:57.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790162+0000", 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000215415 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790377+0000", 2020-03-09T00:24:57.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8597000000000002e-05 2020-03-09T00:24:57.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790396+0000", 2020-03-09T00:24:57.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5052999999999999e-05 2020-03-09T00:24:57.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790421+0000", 2020-03-09T00:24:57.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0005e-05 2020-03-09T00:24:57.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790431+0000", 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6080000000000003e-05 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790507+0000", 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003087078 2020-03-09T00:24:57.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.793594+0000", 2020-03-09T00:24:57.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034322279999999998 2020-03-09T00:24:57.077 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.077 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.077 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.078 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/46 ECSubWrite(tid=28, reqid=client.4751.0:267, at_version=47'12, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:57.078 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.790517+0000", 2020-03-09T00:24:57.078 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.17289378899999999, 2020-03-09T00:24:57.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066761829999999996, 2020-03-09T00:24:57.078 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.079 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.079 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790517+0000", 2020-03-09T00:24:57.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790517+0000", 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5139000000000001e-05 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.080 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790582+0000", 2020-03-09T00:24:57.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8779999999999999e-05 2020-03-09T00:24:57.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790611+0000", 2020-03-09T00:24:57.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.63e-05 2020-03-09T00:24:57.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790627+0000", 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3439999999999997e-06 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790633+0000", 2020-03-09T00:24:57.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00025737500000000001 2020-03-09T00:24:57.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790891+0000", 2020-03-09T00:24:57.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4008e-05 2020-03-09T00:24:57.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790905+0000", 2020-03-09T00:24:57.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0062585979999999998 2020-03-09T00:24:57.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797163+0000", 2020-03-09T00:24:57.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9638999999999999e-05 2020-03-09T00:24:57.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797193+0000", 2020-03-09T00:24:57.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066761829999999996 2020-03-09T00:24:57.086 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.086 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.086 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.087 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/46 ECSubWrite(tid=29, reqid=client.4751.0:269, at_version=47'14, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:57.087 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.791473+0000", 2020-03-09T00:24:57.087 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.17193792499999999, 2020-03-09T00:24:57.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0060858580000000004, 2020-03-09T00:24:57.087 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.087 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.088 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.791473+0000", 2020-03-09T00:24:57.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.791473+0000", 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00051370400000000003 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.791986+0000", 2020-03-09T00:24:57.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5383000000000001e-05 2020-03-09T00:24:57.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.792012+0000", 2020-03-09T00:24:57.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7652999999999999e-05 2020-03-09T00:24:57.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.792029+0000", 2020-03-09T00:24:57.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3429999999999996e-06 2020-03-09T00:24:57.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.792038+0000", 2020-03-09T00:24:57.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017619090000000001 2020-03-09T00:24:57.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.092 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.793800+0000", 2020-03-09T00:24:57.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1706999999999996e-05 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.793871+0000", 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036346320000000001 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797506+0000", 2020-03-09T00:24:57.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2527e-05 2020-03-09T00:24:57.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797558+0000", 2020-03-09T00:24:57.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0060858580000000004 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/46 ECSubWrite(tid=30, reqid=client.4751.0:270, at_version=47'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.796010+0000", 2020-03-09T00:24:57.095 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.16740064499999999, 2020-03-09T00:24:57.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044355050000000002, 2020-03-09T00:24:57.096 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.097 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.097 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.796010+0000", 2020-03-09T00:24:57.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.098 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.796010+0000", 2020-03-09T00:24:57.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000158095 2020-03-09T00:24:57.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.796168+0000", 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3164e-05 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.796191+0000", 2020-03-09T00:24:57.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7533e-05 2020-03-09T00:24:57.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.796209+0000", 2020-03-09T00:24:57.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6200000000000001e-06 2020-03-09T00:24:57.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.796215+0000", 2020-03-09T00:24:57.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0016725729999999999 2020-03-09T00:24:57.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797888+0000", 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5579000000000001e-05 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797903+0000", 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0024960289999999999 2020-03-09T00:24:57.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.800399+0000", 2020-03-09T00:24:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5911999999999997e-05 2020-03-09T00:24:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.800445+0000", 2020-03-09T00:24:57.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0044355050000000002 2020-03-09T00:24:57.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.104 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.104 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(3.6s1 47/46 [PushOp(3:6d582d75:::smithi10428818-10:head, version: 35'10, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:6d582d75:::smithi10428818-10:head@35'10, size: 1048576, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.797742+0000", 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.16566838, 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051972290000000003, 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797742+0000", 2020-03-09T00:24:57.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.797742+0000", 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00028132400000000001 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.798023+0000", 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9494e-05 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.798043+0000", 2020-03-09T00:24:57.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8645000000000001e-05 2020-03-09T00:24:57.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.798072+0000", 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0308e-05 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.798082+0000", 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2921000000000003e-05 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.798145+0000", 2020-03-09T00:24:57.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047945369999999998 2020-03-09T00:24:57.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.802939+0000", 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051972290000000003 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(3.3s0 47/46 [PushOp(3:c607ec57:::smithi10428818-30:head, version: 35'6, data_included: [0~122880], data_size: 122880, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:c607ec57:::smithi10428818-30:head@35'6, size: 491520, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:491520, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:24:57.111 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.807188+0000", 2020-03-09T00:24:57.112 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.15622238699999999, 2020-03-09T00:24:57.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0033605589999999999, 2020-03-09T00:24:57.112 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.112 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.112 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.807188+0000", 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.807188+0000", 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00024906700000000001 2020-03-09T00:24:57.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.807437+0000", 2020-03-09T00:24:57.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1554000000000001e-05 2020-03-09T00:24:57.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.807459+0000", 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9848999999999998e-05 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.807489+0000", 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2724999999999999e-05 2020-03-09T00:24:57.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.807501+0000", 2020-03-09T00:24:57.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9399000000000004e-05 2020-03-09T00:24:57.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.807571+0000", 2020-03-09T00:24:57.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002977965 2020-03-09T00:24:57.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.810549+0000", 2020-03-09T00:24:57.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0033605589999999999 2020-03-09T00:24:57.119 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.119 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.119 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.119 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(3.3s0 47/46 [PushOp(3:c7078095:::smithi10428818-50:4, version: 38'10, data_included: [0~208896], data_size: 208896, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:c7078095:::smithi10428818-50:4@38'10, size: 835584, copy_subset: [], clone_subset: {}, snapset: 4=[]:{4=[4]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:835584, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:24:57.120 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.814262+0000", 2020-03-09T00:24:57.120 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.14914860099999999, 2020-03-09T00:24:57.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003889071, 2020-03-09T00:24:57.120 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.120 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.120 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.814262+0000", 2020-03-09T00:24:57.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.814262+0000", 2020-03-09T00:24:57.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00027184300000000002 2020-03-09T00:24:57.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.814534+0000", 2020-03-09T00:24:57.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4839000000000001e-05 2020-03-09T00:24:57.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.123 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.814559+0000", 2020-03-09T00:24:57.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2110000000000003e-05 2020-03-09T00:24:57.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.814591+0000", 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8632000000000001e-05 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.814609+0000", 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4918000000000002e-05 2020-03-09T00:24:57.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.814704+0000", 2020-03-09T00:24:57.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034467289999999999 2020-03-09T00:24:57.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.818151+0000", 2020-03-09T00:24:57.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003889071 2020-03-09T00:24:57.126 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.126 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.126 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.127 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDPGPush(3.3s0 47/46 [PushOp(3:c7078095:::smithi10428818-50:head, version: 38'13, data_included: [0~208896], data_size: 208896, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:c7078095:::smithi10428818-50:head@38'13, size: 835584, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:835584, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:24:57.127 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.821500+0000", 2020-03-09T00:24:57.127 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.14191002999999999, 2020-03-09T00:24:57.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036647170000000001, 2020-03-09T00:24:57.127 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:57.127 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.127 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.821500+0000", 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:57.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.821500+0000", 2020-03-09T00:24:57.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00027049099999999998 2020-03-09T00:24:57.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:57.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.821771+0000", 2020-03-09T00:24:57.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8312999999999999e-05 2020-03-09T00:24:57.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:57.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.821789+0000", 2020-03-09T00:24:57.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0561000000000001e-05 2020-03-09T00:24:57.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:57.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.821820+0000", 2020-03-09T00:24:57.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2955e-05 2020-03-09T00:24:57.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.821833+0000", 2020-03-09T00:24:57.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0671000000000002e-05 2020-03-09T00:24:57.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:57.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.821903+0000", 2020-03-09T00:24:57.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032617259999999999 2020-03-09T00:24:57.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:57.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:57.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:57.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.825165+0000", 2020-03-09T00:24:57.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036647170000000001 2020-03-09T00:24:57.133 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.133 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.134 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.134 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:57.134 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:57.134 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:57.136 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:57.136 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:57.136 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:57.137 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:57.137 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:24:57.137 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:24:57.137 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:57.074+0000 7fafd77fe700 0 -- 172.21.15.148:0/1928070243 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7fafd818b260 msgr2=0x7fafd818d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-03-09T00:24:57.137 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:24:57.134+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:24:57.138 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:24:57.236 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:57.234+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:57.257 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:57.257 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:57.257 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:24:57.257 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:57.265 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:24:57.297 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:24:57.298 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.298 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.298 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.298 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.299 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.299 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.299 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.299 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.299 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.299 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.299 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.300 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.300 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.300 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.300 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.300 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:24:57.336 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:57.335+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:57.380 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:57.380 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:57.380 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:24:57.381 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:24:57.381 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:57.388 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:24:57.438 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:57.436+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:57.505 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:57.505 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:24:57.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:24:57.505 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:24:57.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.505 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 38/32 ECSubWrite(tid=32, reqid=client.4751.0:187, at_version=38'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:57.506 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.817769+0000", 2020-03-09T00:24:57.506 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6843751820000001, 2020-03-09T00:24:57.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015698909999999999, 2020-03-09T00:24:57.506 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.507 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.507 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817769+0000", 2020-03-09T00:24:57.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817769+0000", 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7283000000000002e-05 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817806+0000", 2020-03-09T00:24:57.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9885e-05 2020-03-09T00:24:57.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817826+0000", 2020-03-09T00:24:57.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.3850000000000008e-06 2020-03-09T00:24:57.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817835+0000", 2020-03-09T00:24:57.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6299999999999998e-06 2020-03-09T00:24:57.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817843+0000", 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2272999999999999e-05 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817895+0000", 2020-03-09T00:24:57.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2044e-05 2020-03-09T00:24:57.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817907+0000", 2020-03-09T00:24:57.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001383472 2020-03-09T00:24:57.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819291+0000", 2020-03-09T00:24:57.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7919e-05 2020-03-09T00:24:57.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819339+0000", 2020-03-09T00:24:57.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015698909999999999 2020-03-09T00:24:57.517 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.517 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.517 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 38/32 ECSubWrite(tid=33, reqid=client.4751.0:188, at_version=38'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.818488+0000", 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6836562280000003, 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001272544, 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.518 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818488+0000", 2020-03-09T00:24:57.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818488+0000", 2020-03-09T00:24:57.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1846000000000004e-05 2020-03-09T00:24:57.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818550+0000", 2020-03-09T00:24:57.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0030999999999998e-05 2020-03-09T00:24:57.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818570+0000", 2020-03-09T00:24:57.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5673000000000001e-05 2020-03-09T00:24:57.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818585+0000", 2020-03-09T00:24:57.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0786000000000001e-05 2020-03-09T00:24:57.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818606+0000", 2020-03-09T00:24:57.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00017353799999999999 2020-03-09T00:24:57.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818780+0000", 2020-03-09T00:24:57.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9639e-05 2020-03-09T00:24:57.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818799+0000", 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093138100000000001 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819731+0000", 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9649999999999999e-05 2020-03-09T00:24:57.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819760+0000", 2020-03-09T00:24:57.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001272544 2020-03-09T00:24:57.528 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.528 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.528 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.529 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 38/32 ECSubWrite(tid=34, reqid=client.4751.0:189, at_version=38'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:57.529 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.820246+0000", 2020-03-09T00:24:57.529 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6818979659999993, 2020-03-09T00:24:57.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050900499999999998, 2020-03-09T00:24:57.529 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.530 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.530 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820246+0000", 2020-03-09T00:24:57.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820246+0000", 2020-03-09T00:24:57.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0267999999999999e-05 2020-03-09T00:24:57.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820316+0000", 2020-03-09T00:24:57.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9732e-05 2020-03-09T00:24:57.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820336+0000", 2020-03-09T00:24:57.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8408000000000001e-05 2020-03-09T00:24:57.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820354+0000", 2020-03-09T00:24:57.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1090000000000003e-06 2020-03-09T00:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820361+0000", 2020-03-09T00:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5114999999999998e-05 2020-03-09T00:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820396+0000", 2020-03-09T00:24:57.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1530000000000002e-06 2020-03-09T00:24:57.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820402+0000", 2020-03-09T00:24:57.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029930100000000003 2020-03-09T00:24:57.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820701+0000", 2020-03-09T00:24:57.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3919000000000003e-05 2020-03-09T00:24:57.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.820755+0000", 2020-03-09T00:24:57.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050900499999999998 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 38/32 ECSubWrite(tid=35, reqid=client.4751.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.825665+0000", 2020-03-09T00:24:57.538 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6764794310000006, 2020-03-09T00:24:57.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00063254399999999997, 2020-03-09T00:24:57.539 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.539 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.539 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825665+0000", 2020-03-09T00:24:57.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825665+0000", 2020-03-09T00:24:57.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0074e-05 2020-03-09T00:24:57.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825695+0000", 2020-03-09T00:24:57.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.048e-05 2020-03-09T00:24:57.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825705+0000", 2020-03-09T00:24:57.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5277999999999998e-05 2020-03-09T00:24:57.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825720+0000", 2020-03-09T00:24:57.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1060000000000006e-06 2020-03-09T00:24:57.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825730+0000", 2020-03-09T00:24:57.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6544e-05 2020-03-09T00:24:57.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825776+0000", 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1715e-05 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825788+0000", 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000460839 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826249+0000", 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8507999999999997e-05 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826297+0000", 2020-03-09T00:24:57.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00063254399999999997 2020-03-09T00:24:57.546 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.546 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.546 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 38/32 ECSubWrite(tid=36, reqid=client.4751.0:204, at_version=38'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.839116+0000", 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6630278989999994, 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024354519999999998, 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.547 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839116+0000", 2020-03-09T00:24:57.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839116+0000", 2020-03-09T00:24:57.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0892999999999997e-05 2020-03-09T00:24:57.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839147+0000", 2020-03-09T00:24:57.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5358000000000001e-05 2020-03-09T00:24:57.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839162+0000", 2020-03-09T00:24:57.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5499999999999995e-06 2020-03-09T00:24:57.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839171+0000", 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839177+0000", 2020-03-09T00:24:57.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9580000000000003e-05 2020-03-09T00:24:57.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839227+0000", 2020-03-09T00:24:57.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3915e-05 2020-03-09T00:24:57.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.839240+0000", 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022639589999999999 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841504+0000", 2020-03-09T00:24:57.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7095999999999998e-05 2020-03-09T00:24:57.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841552+0000", 2020-03-09T00:24:57.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0024354519999999998 2020-03-09T00:24:57.555 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.555 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.555 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.555 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 38/32 ECSubWrite(tid=37, reqid=client.4751.0:205, at_version=38'12, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:57.555 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.841433+0000", 2020-03-09T00:24:57.556 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6607113239999993, 2020-03-09T00:24:57.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001252108, 2020-03-09T00:24:57.556 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.556 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.556 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841433+0000", 2020-03-09T00:24:57.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841433+0000", 2020-03-09T00:24:57.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8526999999999997e-05 2020-03-09T00:24:57.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841491+0000", 2020-03-09T00:24:57.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1481e-05 2020-03-09T00:24:57.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841513+0000", 2020-03-09T00:24:57.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6697e-05 2020-03-09T00:24:57.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841529+0000", 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0549999999999996e-06 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841535+0000", 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6539000000000002e-05 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841592+0000", 2020-03-09T00:24:57.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5565000000000001e-05 2020-03-09T00:24:57.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841608+0000", 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010337790000000001 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.842641+0000", 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3464999999999998e-05 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.842685+0000", 2020-03-09T00:24:57.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001252108 2020-03-09T00:24:57.563 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.563 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.563 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.564 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 38/32 ECSubWrite(tid=38, reqid=client.4751.0:206, at_version=38'13, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:57.564 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.843214+0000", 2020-03-09T00:24:57.564 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6589301840000008, 2020-03-09T00:24:57.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047067099999999998, 2020-03-09T00:24:57.564 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.565 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843214+0000", 2020-03-09T00:24:57.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843214+0000", 2020-03-09T00:24:57.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3386000000000006e-05 2020-03-09T00:24:57.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843277+0000", 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9443999999999999e-05 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843297+0000", 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7249e-05 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843314+0000", 2020-03-09T00:24:57.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6320000000000008e-06 2020-03-09T00:24:57.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843322+0000", 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7867999999999997e-05 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843359+0000", 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0438999999999999e-05 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843370+0000", 2020-03-09T00:24:57.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00026738100000000002 2020-03-09T00:24:57.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843637+0000", 2020-03-09T00:24:57.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7271999999999997e-05 2020-03-09T00:24:57.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.843684+0000", 2020-03-09T00:24:57.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00047067099999999998 2020-03-09T00:24:57.572 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 38/32 ECSubWrite(tid=39, reqid=client.4751.0:206, at_version=0'0, trim_to=0'0, roll_forward_to=38'13))", 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.846126+0000", 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6560184450000008, 2020-03-09T00:24:57.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072488999999999995, 2020-03-09T00:24:57.574 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.574 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.574 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846126+0000", 2020-03-09T00:24:57.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846126+0000", 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9317000000000001e-05 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846155+0000", 2020-03-09T00:24:57.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4200000000000007e-06 2020-03-09T00:24:57.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846163+0000", 2020-03-09T00:24:57.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.995e-06 2020-03-09T00:24:57.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846173+0000", 2020-03-09T00:24:57.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3380000000000004e-06 2020-03-09T00:24:57.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846179+0000", 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2698999999999999e-05 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846221+0000", 2020-03-09T00:24:57.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1792e-05 2020-03-09T00:24:57.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846233+0000", 2020-03-09T00:24:57.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00056541299999999996 2020-03-09T00:24:57.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846799+0000", 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1916000000000002e-05 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.846850+0000", 2020-03-09T00:24:57.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00072488999999999995 2020-03-09T00:24:57.581 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.581 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.581 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.581 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:207 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 5] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:24:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6452257039999996, 2020-03-09T00:24:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001, 2020-03-09T00:24:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:24:57.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:24:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7176999999999996e-05 2020-03-09T00:24:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856995+0000", 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7161999999999999e-05 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857013+0000", 2020-03-09T00:24:57.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2879e-05 2020-03-09T00:24:57.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857026+0000", 2020-03-09T00:24:57.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6669999999999996e-06 2020-03-09T00:24:57.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857033+0000", 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6767999999999997e-05 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857080+0000", 2020-03-09T00:24:57.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4165999999999994e-05 2020-03-09T00:24:57.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:57.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857154+0000", 2020-03-09T00:24:57.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068311889999999997 2020-03-09T00:24:57.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.863985+0000", 2020-03-09T00:24:57.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041278249999999999 2020-03-09T00:24:57.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868113+0000", 2020-03-09T00:24:57.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001971132 2020-03-09T00:24:57.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:24:57.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870084+0000", 2020-03-09T00:24:57.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113266 2020-03-09T00:24:57.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870198+0000", 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:208 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6450559580000004, 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001, 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:24:57.592 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:24:57.593 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:24:57.593 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:57.593 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 208 2020-03-09T00:24:57.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.593 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5717000000000003e-05 2020-03-09T00:24:57.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857134+0000", 2020-03-09T00:24:57.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3659999999999997e-06 2020-03-09T00:24:57.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857140+0000", 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2546e-05 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857153+0000", 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7190000000000001e-06 2020-03-09T00:24:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857157+0000", 2020-03-09T00:24:57.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134509 2020-03-09T00:24:57.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857292+0000", 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5519999999999999e-05 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857327+0000", 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010666132 2020-03-09T00:24:57.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867994+0000", 2020-03-09T00:24:57.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110682 2020-03-09T00:24:57.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:24:57.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868104+0000", 2020-03-09T00:24:57.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00321515 2020-03-09T00:24:57.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871319+0000", 2020-03-09T00:24:57.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011457 2020-03-09T00:24:57.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:24:57.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871434+0000", 2020-03-09T00:24:57.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013121699999999999 2020-03-09T00:24:57.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871565+0000", 2020-03-09T00:24:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001 2020-03-09T00:24:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 38/32 ECSubWriteReply(tid=5, last_complete=38'4, committed=1, applied=1))", 2020-03-09T00:24:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.867888+0000", 2020-03-09T00:24:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6342555450000003, 2020-03-09T00:24:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030145299999999997, 2020-03-09T00:24:57.603 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867888+0000", 2020-03-09T00:24:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867888+0000", 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0086e-05 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867949+0000", 2020-03-09T00:24:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8100000000000004e-06 2020-03-09T00:24:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867957+0000", 2020-03-09T00:24:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4501000000000001e-05 2020-03-09T00:24:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867972+0000", 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8819999999999998e-06 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867978+0000", 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019743500000000001 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868175+0000", 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4739000000000001e-05 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868190+0000", 2020-03-09T00:24:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030145299999999997 2020-03-09T00:24:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.as0 38/32 ECSubWriteReply(tid=5, last_complete=38'4, committed=1, applied=1))", 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.869883+0000", 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6322608499999998, 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001037079, 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.610 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869883+0000", 2020-03-09T00:24:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869883+0000", 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0148999999999999e-05 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869913+0000", 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5329999999999996e-06 2020-03-09T00:24:57.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869918+0000", 2020-03-09T00:24:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.7290000000000006e-06 2020-03-09T00:24:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869927+0000", 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5650000000000003e-06 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869931+0000", 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9589999999999999e-05 2020-03-09T00:24:57.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869971+0000", 2020-03-09T00:24:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094951300000000003 2020-03-09T00:24:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870920+0000", 2020-03-09T00:24:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001037079 2020-03-09T00:24:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/32 ECSubWrite(tid=42, reqid=osd.13.0:42, at_version=42'9, trim_to=0'0, roll_forward_to=41'7))", 2020-03-09T00:24:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.854419+0000", 2020-03-09T00:24:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.6477252890000003, 2020-03-09T00:24:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010130720000000001, 2020-03-09T00:24:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854419+0000", 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854419+0000", 2020-03-09T00:24:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0058e-05 2020-03-09T00:24:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854479+0000", 2020-03-09T00:24:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2490999999999999e-05 2020-03-09T00:24:57.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854501+0000", 2020-03-09T00:24:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3159000000000001e-05 2020-03-09T00:24:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854514+0000", 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4959999999999999e-06 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854522+0000", 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6814000000000003e-05 2020-03-09T00:24:57.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854569+0000", 2020-03-09T00:24:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.892e-06 2020-03-09T00:24:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.854577+0000", 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000824525 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855401+0000", 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0636999999999998e-05 2020-03-09T00:24:57.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.855432+0000", 2020-03-09T00:24:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010130720000000001 2020-03-09T00:24:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.624 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 42/32 ECSubWrite(tid=43, reqid=osd.13.0:42, at_version=0'0, trim_to=0'0, roll_forward_to=42'9))", 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.859933+0000", 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.6422106789999997, 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014992650000000001, 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.625 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.626 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859933+0000", 2020-03-09T00:24:57.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859933+0000", 2020-03-09T00:24:57.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.99e-05 2020-03-09T00:24:57.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859983+0000", 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7440000000000002e-06 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859993+0000", 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1330000000000005e-06 2020-03-09T00:24:57.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.860002+0000", 2020-03-09T00:24:57.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2519999999999996e-06 2020-03-09T00:24:57.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.860008+0000", 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8510000000000001e-05 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.860057+0000", 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3210999999999999e-05 2020-03-09T00:24:57.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.860070+0000", 2020-03-09T00:24:57.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011352370000000001 2020-03-09T00:24:57.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.861205+0000", 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000227278 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.861433+0000", 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014992650000000001 2020-03-09T00:24:57.632 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.633 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.633 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.633 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 44/32 ECSubWrite(tid=44, reqid=osd.13.0:44, at_version=44'11, trim_to=0'0, roll_forward_to=42'9))", 2020-03-09T00:24:57.633 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:24:57.633 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.6530027270000001, 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002, 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:24:57.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:24:57.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8740999999999999e-05 2020-03-09T00:24:57.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849200+0000", 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2819e-05 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849223+0000", 2020-03-09T00:24:57.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5366000000000002e-05 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849238+0000", 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7410000000000003e-06 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849245+0000", 2020-03-09T00:24:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5445999999999996e-05 2020-03-09T00:24:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849310+0000", 2020-03-09T00:24:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3651e-05 2020-03-09T00:24:57.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849324+0000", 2020-03-09T00:24:57.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057096769999999998 2020-03-09T00:24:57.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855034+0000", 2020-03-09T00:24:57.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001136 2020-03-09T00:24:57.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855147+0000", 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.641 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:57.642 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:24:57.642 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.651861142, 2020-03-09T00:24:57.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998, 2020-03-09T00:24:57.642 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.642 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.642 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:24:57.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018406799999999999 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850467+0000", 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2490999999999999e-05 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850489+0000", 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6807999999999999e-05 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9419999999999995e-06 2020-03-09T00:24:57.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850514+0000", 2020-03-09T00:24:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088484100000000003 2020-03-09T00:24:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851399+0000", 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2279999999999999e-05 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851411+0000", 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083268249999999995 2020-03-09T00:24:57.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859738+0000", 2020-03-09T00:24:57.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2222999999999998e-05 2020-03-09T00:24:57.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859790+0000", 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.649 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:57.650 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.856437+0000", 2020-03-09T00:24:57.650 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.6457067580000002, 2020-03-09T00:24:57.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037319300000000001, 2020-03-09T00:24:57.650 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.650 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.650 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856437+0000", 2020-03-09T00:24:57.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856437+0000", 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024651700000000003 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856684+0000", 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8229000000000001e-05 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856702+0000", 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4171e-05 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856716+0000", 2020-03-09T00:24:57.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4409999999999998e-06 2020-03-09T00:24:57.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856723+0000", 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000123032 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856846+0000", 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2551999999999999e-05 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.856858+0000", 2020-03-09T00:24:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032676229999999999 2020-03-09T00:24:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.860126+0000", 2020-03-09T00:24:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3365000000000002e-05 2020-03-09T00:24:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.860169+0000", 2020-03-09T00:24:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037319300000000001 2020-03-09T00:24:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 44/32 ECSubWrite(tid=45, reqid=osd.13.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=44'11))", 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.857918+0000", 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.64422567, 2020-03-09T00:24:57.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00043500800000000002, 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857918+0000", 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857918+0000", 2020-03-09T00:24:57.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4524999999999997e-05 2020-03-09T00:24:57.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857973+0000", 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2156e-05 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857985+0000", 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1603999999999999e-05 2020-03-09T00:24:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857997+0000", 2020-03-09T00:24:57.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3780000000000003e-06 2020-03-09T00:24:57.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.858001+0000", 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6159999999999999e-05 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.858047+0000", 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2320000000000001e-05 2020-03-09T00:24:57.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.858059+0000", 2020-03-09T00:24:57.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00024746700000000002 2020-03-09T00:24:57.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.858307+0000", 2020-03-09T00:24:57.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6397999999999999e-05 2020-03-09T00:24:57.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.858353+0000", 2020-03-09T00:24:57.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00043500800000000002 2020-03-09T00:24:57.666 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.666 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.666 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.666 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=39, reqid=client.4751.0:244, at_version=44'18, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:57.667 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.862778+0000", 2020-03-09T00:24:57.667 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.6393663530000002, 2020-03-09T00:24:57.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000463714, 2020-03-09T00:24:57.667 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.667 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.667 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862778+0000", 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862778+0000", 2020-03-09T00:24:57.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5850999999999999e-05 2020-03-09T00:24:57.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862813+0000", 2020-03-09T00:24:57.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5169e-05 2020-03-09T00:24:57.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862829+0000", 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2110000000000004e-06 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862838+0000", 2020-03-09T00:24:57.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6999999999999999e-06 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862843+0000", 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8773999999999998e-05 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862891+0000", 2020-03-09T00:24:57.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2534000000000001e-05 2020-03-09T00:24:57.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.862904+0000", 2020-03-09T00:24:57.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029347999999999999 2020-03-09T00:24:57.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.863197+0000", 2020-03-09T00:24:57.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3995000000000001e-05 2020-03-09T00:24:57.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.863241+0000", 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000463714 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.675 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=40, reqid=client.4751.0:244, at_version=0'0, trim_to=0'0, roll_forward_to=44'18))", 2020-03-09T00:24:57.675 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.869732+0000", 2020-03-09T00:24:57.675 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.632412408, 2020-03-09T00:24:57.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057905499999999998, 2020-03-09T00:24:57.675 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:24:57.675 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:24:57.675 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869732+0000", 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:24:57.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869732+0000", 2020-03-09T00:24:57.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6338e-05 2020-03-09T00:24:57.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:24:57.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869808+0000", 2020-03-09T00:24:57.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7601999999999999e-05 2020-03-09T00:24:57.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:24:57.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869826+0000", 2020-03-09T00:24:57.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9437e-05 2020-03-09T00:24:57.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:24:57.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869845+0000", 2020-03-09T00:24:57.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0828e-05 2020-03-09T00:24:57.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:24:57.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869856+0000", 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7051e-05 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869913+0000", 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5671999999999998e-05 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:24:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.869929+0000", 2020-03-09T00:24:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032732600000000002 2020-03-09T00:24:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:24:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.870256+0000", 2020-03-09T00:24:57.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4800999999999999e-05 2020-03-09T00:24:57.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:24:57.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:24:57.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:24:57.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.870311+0000", 2020-03-09T00:24:57.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057905499999999998 2020-03-09T00:24:57.682 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.683 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.683 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.683 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:24:57.683 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:24:57.683 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:57.685 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:24:57.685 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:57.683+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:57.772 INFO:tasks.rados.rados.0.smithi104.stdout:107: done (3 left) 2020-03-09T00:24:57.773 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 31 current snap is 8 2020-03-09T00:24:57.773 INFO:tasks.rados.rados.0.smithi104.stdout:108: seq_num 64 ranges {1441792=704512,2146304=376832} 2020-03-09T00:24:57.784 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is unset 2020-03-09T00:24:57.785 INFO:tasks.rados.rados.0.smithi104.stdout:108: writing smithi10428818-31 from 1441792 to 2146304 tid 1 2020-03-09T00:24:57.786 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:57.784+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:57.788 INFO:tasks.rados.rados.0.smithi104.stdout:108: writing smithi10428818-31 from 2146304 to 2523136 tid 2 2020-03-09T00:24:57.788 INFO:tasks.rados.rados.0.smithi104.stdout:109: copy_from oid 46 from oid 30 current snap is 8 2020-03-09T00:24:57.789 INFO:tasks.rados.rados.0.smithi104.stdout:104: finishing write tid 1 to smithi10428818-19 2020-03-09T00:24:57.789 INFO:tasks.rados.rados.0.smithi104.stdout:104: finishing write tid 2 to smithi10428818-19 2020-03-09T00:24:57.789 INFO:tasks.rados.rados.0.smithi104.stdout:104: finishing write tid 4 to smithi10428818-19 2020-03-09T00:24:57.789 INFO:tasks.rados.rados.0.smithi104.stdout:105: finishing write tid 1 to smithi10428818-12 2020-03-09T00:24:57.789 INFO:tasks.rados.rados.0.smithi104.stdout:105: finishing write tid 2 to smithi10428818-12 2020-03-09T00:24:57.789 INFO:tasks.rados.rados.0.smithi104.stdout:105: finishing write tid 4 to smithi10428818-12 2020-03-09T00:24:57.789 INFO:tasks.rados.rados.0.smithi104.stdout:104: finishing write tid 5 to smithi10428818-19 2020-03-09T00:24:57.790 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 19 v 12 (ObjNum 62 snap 8 seq_num 62) dirty exists 2020-03-09T00:24:57.790 INFO:tasks.rados.rados.0.smithi104.stdout:104: left oid 19 (ObjNum 62 snap 8 seq_num 62) 2020-03-09T00:24:57.790 INFO:tasks.rados.rados.0.smithi104.stdout:105: finishing write tid 5 to smithi10428818-12 2020-03-09T00:24:57.790 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 12 v 16 (ObjNum 63 snap 8 seq_num 63) dirty exists 2020-03-09T00:24:57.790 INFO:tasks.rados.rados.0.smithi104.stdout:105: left oid 12 (ObjNum 63 snap 8 seq_num 63) 2020-03-09T00:24:57.790 INFO:tasks.rados.rados.0.smithi104.stdout:102: finishing copy_from to smithi10428818-34 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 34 v 8 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:102: finishing copy_from racing read to smithi10428818-34 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:102: done (4 left) 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:104: done (3 left) 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:105: done (2 left) 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:110: rmattr oid 49 current snap is 8 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:110: done (2 left) 2020-03-09T00:24:57.791 INFO:tasks.rados.rados.0.smithi104.stdout:111: rmattr oid 39 current snap is 8 2020-03-09T00:24:57.792 INFO:tasks.rados.rados.0.smithi104.stdout:111: done (2 left) 2020-03-09T00:24:57.792 INFO:tasks.rados.rados.0.smithi104.stdout:112: snap_create 2020-03-09T00:24:57.798 INFO:tasks.rados.rados.0.smithi104.stdout:108: finishing write tid 1 to smithi10428818-31 2020-03-09T00:24:57.798 INFO:tasks.rados.rados.0.smithi104.stdout:108: finishing write tid 2 to smithi10428818-31 2020-03-09T00:24:57.798 INFO:tasks.rados.rados.0.smithi104.stdout:108: finishing write tid 4 to smithi10428818-31 2020-03-09T00:24:57.800 INFO:tasks.rados.rados.0.smithi104.stdout:108: finishing write tid 5 to smithi10428818-31 2020-03-09T00:24:57.800 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 31 v 14 (ObjNum 64 snap 8 seq_num 64) dirty exists 2020-03-09T00:24:57.800 INFO:tasks.rados.rados.0.smithi104.stdout:108: left oid 31 (ObjNum 64 snap 8 seq_num 64) 2020-03-09T00:24:57.822 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:57.822 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:57.823 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:24:57.823 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:57.831 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:24:57.886 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:57.885+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:57.947 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:57.947 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:24:57.947 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:24:57.948 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:24:57.948 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:57.956 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:24:57.988 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:57.986+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:58.074 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:24:58.074 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:24:58.074 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:24:58.075 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:24:58.075 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.075 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/32 ECSubWrite(tid=42, reqid=client.4751.0:192, at_version=38'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:58.075 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.816878+0000", 2020-03-09T00:24:58.075 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.254902985999999, 2020-03-09T00:24:58.075 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015187989999999999, 2020-03-09T00:24:58.075 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.076 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.076 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816878+0000", 2020-03-09T00:24:58.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.076 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816878+0000", 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0727999999999999e-05 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816939+0000", 2020-03-09T00:24:58.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6418999999999999e-05 2020-03-09T00:24:58.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816965+0000", 2020-03-09T00:24:58.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9264999999999999e-05 2020-03-09T00:24:58.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816984+0000", 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7679999999999997e-06 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.816993+0000", 2020-03-09T00:24:58.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1686000000000005e-05 2020-03-09T00:24:58.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817055+0000", 2020-03-09T00:24:58.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3050999999999999e-05 2020-03-09T00:24:58.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.817068+0000", 2020-03-09T00:24:58.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001279266 2020-03-09T00:24:58.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818347+0000", 2020-03-09T00:24:58.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9616000000000001e-05 2020-03-09T00:24:58.082 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818397+0000", 2020-03-09T00:24:58.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015187989999999999 2020-03-09T00:24:58.083 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.083 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.083 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.083 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 38/32 ECSubWrite(tid=32, reqid=client.4751.0:187, at_version=38'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:58.084 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.818035+0000", 2020-03-09T00:24:58.084 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.253745954999999, 2020-03-09T00:24:58.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0021995299999999999, 2020-03-09T00:24:58.084 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.084 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.084 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818035+0000", 2020-03-09T00:24:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.085 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818035+0000", 2020-03-09T00:24:58.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2366999999999994e-05 2020-03-09T00:24:58.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.086 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818097+0000", 2020-03-09T00:24:58.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5174999999999999e-05 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818123+0000", 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5798999999999999e-05 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818138+0000", 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3969999999999997e-06 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818146+0000", 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6965000000000001e-05 2020-03-09T00:24:58.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818203+0000", 2020-03-09T00:24:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6583999999999999e-05 2020-03-09T00:24:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818219+0000", 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001973308 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820193+0000", 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1934999999999997e-05 2020-03-09T00:24:58.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820235+0000", 2020-03-09T00:24:58.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0021995299999999999 2020-03-09T00:24:58.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.091 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.091 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.092 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 38/32 ECSubWrite(tid=33, reqid=client.4751.0:188, at_version=38'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:58.092 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.818613+0000", 2020-03-09T00:24:58.092 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.253168456999999, 2020-03-09T00:24:58.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027746490000000001, 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818613+0000", 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818613+0000", 2020-03-09T00:24:58.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7079e-05 2020-03-09T00:24:58.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818670+0000", 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1268e-05 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818691+0000", 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7980000000000001e-05 2020-03-09T00:24:58.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818709+0000", 2020-03-09T00:24:58.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4419999999999998e-06 2020-03-09T00:24:58.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818717+0000", 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00068360299999999997 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819401+0000", 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.169e-05 2020-03-09T00:24:58.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819413+0000", 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001823872 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821236+0000", 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000150715 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821387+0000", 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027746490000000001 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.099 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/32 ECSubWrite(tid=43, reqid=client.4751.0:193, at_version=38'12, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.818882+0000", 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.252898936999999, 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023957710000000001, 2020-03-09T00:24:58.100 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.101 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.101 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818882+0000", 2020-03-09T00:24:58.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818882+0000", 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8992999999999995e-05 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818951+0000", 2020-03-09T00:24:58.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9978000000000001e-05 2020-03-09T00:24:58.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818971+0000", 2020-03-09T00:24:58.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7170999999999999e-05 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818988+0000", 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2910000000000004e-06 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.818995+0000", 2020-03-09T00:24:58.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014864000000000001 2020-03-09T00:24:58.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819143+0000", 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4830999999999999e-05 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819178+0000", 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020430639999999998 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821221+0000", 2020-03-09T00:24:58.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6802999999999998e-05 2020-03-09T00:24:58.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821278+0000", 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023957710000000001 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.108 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 38/32 ECSubWrite(tid=44, reqid=client.4751.0:194, at_version=38'13, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.819715+0000", 2020-03-09T00:24:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.252065708, 2020-03-09T00:24:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001584137, 2020-03-09T00:24:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819715+0000", 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819715+0000", 2020-03-09T00:24:58.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6344000000000002e-05 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819762+0000", 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9618000000000001e-05 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.111 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819781+0000", 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5311e-05 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819797+0000", 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3463e-05 2020-03-09T00:24:58.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819810+0000", 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2447999999999997e-05 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819852+0000", 2020-03-09T00:24:58.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1772e-05 2020-03-09T00:24:58.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.819864+0000", 2020-03-09T00:24:58.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001416788 2020-03-09T00:24:58.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821281+0000", 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8393000000000002e-05 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821299+0000", 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001584137 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 38/32 ECSubWrite(tid=34, reqid=client.4751.0:189, at_version=38'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:58.116 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.820142+0000", 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.251638852999999, 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001283029, 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820142+0000", 2020-03-09T00:24:58.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820142+0000", 2020-03-09T00:24:58.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6421000000000001e-05 2020-03-09T00:24:58.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820199+0000", 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.855e-05 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820217+0000", 2020-03-09T00:24:58.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.523e-05 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820232+0000", 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9529999999999996e-06 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820238+0000", 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2357000000000001e-05 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820281+0000", 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1715999999999999e-05 2020-03-09T00:24:58.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.820292+0000", 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001099119 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821391+0000", 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3683000000000003e-05 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.821425+0000", 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001283029 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.123 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.124 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.124 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 38/32 ECSubWrite(tid=37, reqid=client.4751.0:205, at_version=38'12, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:58.124 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.841717+0000", 2020-03-09T00:24:58.124 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.23006436, 2020-03-09T00:24:58.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0028421539999999999, 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841717+0000", 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.125 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841717+0000", 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8493999999999998e-05 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841785+0000", 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2527e-05 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841808+0000", 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6623e-05 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841824+0000", 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6309999999999997e-06 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841831+0000", 2020-03-09T00:24:58.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010730000000000001 2020-03-09T00:24:58.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841938+0000", 2020-03-09T00:24:58.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4630000000000004e-06 2020-03-09T00:24:58.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.841946+0000", 2020-03-09T00:24:58.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0025799820000000002 2020-03-09T00:24:58.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844526+0000", 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3133999999999998e-05 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844559+0000", 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0028421539999999999 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.131 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 38/32 ECSubWrite(tid=38, reqid=client.4751.0:206, at_version=38'13, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.843334+0000", 2020-03-09T00:24:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.22844746, 2020-03-09T00:24:58.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013958830000000001, 2020-03-09T00:24:58.133 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.133 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.133 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843334+0000", 2020-03-09T00:24:58.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843334+0000", 2020-03-09T00:24:58.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0716999999999997e-05 2020-03-09T00:24:58.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843364+0000", 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3360000000000001e-05 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843378+0000", 2020-03-09T00:24:58.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0767999999999999e-05 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843388+0000", 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3299999999999998e-06 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.136 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843394+0000", 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1347000000000002e-05 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843435+0000", 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1800000000000002e-06 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.137 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.843444+0000", 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0012554210000000001 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844700+0000", 2020-03-09T00:24:58.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.976e-05 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.844729+0000", 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013958830000000001 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.139 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 43/32 ECSubWrite(tid=48, reqid=client.4751.0:236, at_version=43'15, trim_to=0'0, roll_forward_to=38'13))", 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:52.836772+0000", 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.2350086810000001, 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011830949999999999, 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.140 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836772+0000", 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.142 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836772+0000", 2020-03-09T00:24:58.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.902e-05 2020-03-09T00:24:58.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.142 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.142 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836841+0000", 2020-03-09T00:24:58.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1300999999999997e-05 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836873+0000", 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7895000000000001e-05 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836891+0000", 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0333999999999999e-05 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836911+0000", 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1736999999999995e-05 2020-03-09T00:24:58.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836973+0000", 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4667999999999999e-05 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.836987+0000", 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00091497499999999995 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.837902+0000", 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3164999999999997e-05 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.146 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:52.837955+0000", 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011830949999999999 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.147 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 44/32 ECSubWrite(tid=37, reqid=osd.14.0:37, at_version=44'8, trim_to=0'0, roll_forward_to=42'6))", 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.841586+0000", 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2301947850000001, 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020030809999999999, 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.148 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841586+0000", 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841586+0000", 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4336000000000003e-05 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.150 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841641+0000", 2020-03-09T00:24:58.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6389000000000001e-05 2020-03-09T00:24:58.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.150 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841667+0000", 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7859999999999998e-05 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841685+0000", 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0849999999999999e-06 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841692+0000", 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0887999999999997e-05 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841753+0000", 2020-03-09T00:24:58.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5968e-05 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.841769+0000", 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001784234 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.843553+0000", 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6321e-05 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.843589+0000", 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020030809999999999 2020-03-09T00:24:58.154 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 44/32 ECSubWrite(tid=38, reqid=osd.14.0:37, at_version=0'0, trim_to=0'0, roll_forward_to=44'8))", 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.844152+0000", 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2276294989999998, 2020-03-09T00:24:58.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000943954, 2020-03-09T00:24:58.156 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.156 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.156 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.156 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844152+0000", 2020-03-09T00:24:58.156 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844152+0000", 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3344000000000003e-05 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.157 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844205+0000", 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8229e-05 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844233+0000", 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7414e-05 2020-03-09T00:24:58.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844250+0000", 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1010000000000003e-06 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844258+0000", 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3437999999999999e-05 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844311+0000", 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5235000000000001e-05 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.844326+0000", 2020-03-09T00:24:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00073587499999999998 2020-03-09T00:24:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.845062+0000", 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3318000000000002e-05 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.845095+0000", 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000943954 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.162 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.163 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.163 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:58.163 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:24:58.163 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2204012510000002, 2020-03-09T00:24:58.163 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999, 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:24:58.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017980700000000001 2020-03-09T00:24:58.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851560+0000", 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5363000000000001e-05 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851585+0000", 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4187000000000001e-05 2020-03-09T00:24:58.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851609+0000", 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2890000000000003e-06 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.167 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851618+0000", 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00036370100000000002 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851982+0000", 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3288999999999999e-05 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851995+0000", 2020-03-09T00:24:58.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074675740000000003 2020-03-09T00:24:58.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859463+0000", 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1719999999999999e-05 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859525+0000", 2020-03-09T00:24:58.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999 2020-03-09T00:24:58.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.171 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.171 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.171 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=52, reqid=client.4751.0:246, at_version=44'19, trim_to=0'0, roll_forward_to=44'17))", 2020-03-09T00:24:58.171 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.851696+0000", 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2200852470000001, 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001758597, 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851696+0000", 2020-03-09T00:24:58.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851696+0000", 2020-03-09T00:24:58.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9200999999999998e-05 2020-03-09T00:24:58.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851755+0000", 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.9014000000000001e-05 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851774+0000", 2020-03-09T00:24:58.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3686000000000001e-05 2020-03-09T00:24:58.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851788+0000", 2020-03-09T00:24:58.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5169999999999999e-06 2020-03-09T00:24:58.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851793+0000", 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098895100000000007 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.852782+0000", 2020-03-09T00:24:58.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2408999999999999e-05 2020-03-09T00:24:58.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.852795+0000", 2020-03-09T00:24:58.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00062795099999999996 2020-03-09T00:24:58.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853422+0000", 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1868e-05 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853454+0000", 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001758597 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=53, reqid=client.4751.0:247, at_version=44'20, trim_to=0'0, roll_forward_to=44'17))", 2020-03-09T00:24:58.179 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.853530+0000", 2020-03-09T00:24:58.180 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2182513349999997, 2020-03-09T00:24:58.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00150347, 2020-03-09T00:24:58.180 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.180 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.180 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853530+0000", 2020-03-09T00:24:58.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853530+0000", 2020-03-09T00:24:58.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9698999999999999e-05 2020-03-09T00:24:58.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853589+0000", 2020-03-09T00:24:58.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3281e-05 2020-03-09T00:24:58.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853613+0000", 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7867999999999999e-05 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853631+0000", 2020-03-09T00:24:58.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.9000000000000006e-06 2020-03-09T00:24:58.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853638+0000", 2020-03-09T00:24:58.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9777999999999997e-05 2020-03-09T00:24:58.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853688+0000", 2020-03-09T00:24:58.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3448999999999999e-05 2020-03-09T00:24:58.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.853702+0000", 2020-03-09T00:24:58.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001266826 2020-03-09T00:24:58.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854968+0000", 2020-03-09T00:24:58.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4669e-05 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855033+0000", 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00150347 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=54, reqid=client.4751.0:248, at_version=44'21, trim_to=0'0, roll_forward_to=44'17))", 2020-03-09T00:24:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.854615+0000", 2020-03-09T00:24:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.217166486, 2020-03-09T00:24:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010817629999999999, 2020-03-09T00:24:58.188 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854615+0000", 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854615+0000", 2020-03-09T00:24:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7636999999999997e-05 2020-03-09T00:24:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854672+0000", 2020-03-09T00:24:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6443000000000001e-05 2020-03-09T00:24:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854699+0000", 2020-03-09T00:24:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0339999999999998e-05 2020-03-09T00:24:58.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854719+0000", 2020-03-09T00:24:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2709999999999992e-06 2020-03-09T00:24:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854727+0000", 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2358999999999998e-05 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854780+0000", 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.67e-06 2020-03-09T00:24:58.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.854788+0000", 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00086690200000000002 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855655+0000", 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1140999999999998e-05 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855696+0000", 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010817629999999999 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.195 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 44/32 ECSubWrite(tid=39, reqid=client.4751.0:253, at_version=44'10, trim_to=0'0, roll_forward_to=44'8))", 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.855899+0000", 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2158823879999998, 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033587790000000001, 2020-03-09T00:24:58.196 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855899+0000", 2020-03-09T00:24:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855899+0000", 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2745999999999999e-05 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855961+0000", 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2719000000000001e-05 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855984+0000", 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3929e-05 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.855998+0000", 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3289999999999996e-06 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856005+0000", 2020-03-09T00:24:58.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7617999999999997e-05 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856053+0000", 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2285e-05 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856065+0000", 2020-03-09T00:24:58.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003148086 2020-03-09T00:24:58.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859213+0000", 2020-03-09T00:24:58.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4066999999999997e-05 2020-03-09T00:24:58.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859257+0000", 2020-03-09T00:24:58.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033587790000000001 2020-03-09T00:24:58.203 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.203 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.856599+0000", 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2151819330000002, 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033226729999999999, 2020-03-09T00:24:58.204 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.205 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.205 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856599+0000", 2020-03-09T00:24:58.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856599+0000", 2020-03-09T00:24:58.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013956799999999999 2020-03-09T00:24:58.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856739+0000", 2020-03-09T00:24:58.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6488000000000001e-05 2020-03-09T00:24:58.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856765+0000", 2020-03-09T00:24:58.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6609999999999999e-05 2020-03-09T00:24:58.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856782+0000", 2020-03-09T00:24:58.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2620000000000004e-06 2020-03-09T00:24:58.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856788+0000", 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000167465 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856955+0000", 2020-03-09T00:24:58.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3779e-05 2020-03-09T00:24:58.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.856969+0000", 2020-03-09T00:24:58.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0029193420000000001 2020-03-09T00:24:58.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859889+0000", 2020-03-09T00:24:58.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3158999999999999e-05 2020-03-09T00:24:58.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859922+0000", 2020-03-09T00:24:58.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033226729999999999 2020-03-09T00:24:58.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.212 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.212 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=56, reqid=client.4751.0:254, at_version=44'23, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.2080802439999996, 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996, 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:24:58.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:24:58.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015903600000000001 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863860+0000", 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3975999999999999e-05 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863884+0000", 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6139999999999998e-06 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863893+0000", 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3190000000000002e-06 2020-03-09T00:24:58.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863899+0000", 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039154710000000002 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867814+0000", 2020-03-09T00:24:58.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5379999999999998e-05 2020-03-09T00:24:58.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867830+0000", 2020-03-09T00:24:58.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001627606 2020-03-09T00:24:58.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869457+0000", 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7565999999999998e-05 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869505+0000", 2020-03-09T00:24:58.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996 2020-03-09T00:24:58.220 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.220 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.220 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.220 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 48/32 ECSubWrite(tid=43, reqid=osd.14.0:43, at_version=48'12, trim_to=0'0, roll_forward_to=44'10))", 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:57.769949+0000", 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.30183230999999999, 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010027949999999999, 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.769949+0000", 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.769949+0000", 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0608e-05 2020-03-09T00:24:58.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770009+0000", 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9708000000000001e-05 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770039+0000", 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5801e-05 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770055+0000", 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2876e-05 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770068+0000", 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010246099999999999 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770170+0000", 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4146e-05 2020-03-09T00:24:58.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770184+0000", 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00071622500000000002 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770901+0000", 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0970000000000002e-05 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.770951+0000", 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010027949999999999 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.227 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 48/32 ECSubWrite(tid=42, reqid=osd.8.0:42, at_version=48'20, trim_to=0'0, roll_forward_to=44'18))", 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:57.777204+0000", 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.29457651600000001, 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001197362, 2020-03-09T00:24:58.228 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:24:58.229 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.229 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:24:58.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:24:58.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777204+0000", 2020-03-09T00:24:58.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:24:58.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777204+0000", 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8566000000000001e-05 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:24:58.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777263+0000", 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3626999999999999e-05 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777287+0000", 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5252000000000001e-05 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777302+0000", 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7240000000000003e-06 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777309+0000", 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014193300000000001 2020-03-09T00:24:58.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:24:58.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777451+0000", 2020-03-09T00:24:58.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.322e-05 2020-03-09T00:24:58.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.777464+0000", 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000891601 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.778355+0000", 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6439e-05 2020-03-09T00:24:58.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:57.778402+0000", 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001197362 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.235 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:24:58.236 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:24:58.236 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:24:58.237 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:24:58.238 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:58.236+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:58.338 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:58.336+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:58.374 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:58.407 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:58.407 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:24:58.408 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:58.409 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:24:58.439 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:58.437+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:58.533 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:24:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:24:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:24:58.534 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:58.539 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:58.538+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:58.542 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:24:58.641 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:24:58.639+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:24:58.668 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:24:58.668 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:24:58.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:24:58.668 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:24:58.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.668 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:200 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~557056] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:58.669 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:24:58.669 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.830461886, 2020-03-09T00:24:58.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845, 2020-03-09T00:24:58.669 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.669 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:58.669 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 200 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:24:58.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:24:58.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001017256 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835741+0000", 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0451999999999999e-05 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835751+0000", 2020-03-09T00:24:58.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1429000000000001e-05 2020-03-09T00:24:58.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835773+0000", 2020-03-09T00:24:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0686000000000001e-05 2020-03-09T00:24:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835783+0000", 2020-03-09T00:24:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7098e-05 2020-03-09T00:24:58.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835850+0000", 2020-03-09T00:24:58.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100177 2020-03-09T00:24:58.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835951+0000", 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014653610000000001 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837416+0000", 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082709510000000003 2020-03-09T00:24:58.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845687+0000", 2020-03-09T00:24:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015616149999999999 2020-03-09T00:24:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:58.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847248+0000", 2020-03-09T00:24:58.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6819999999999998e-05 2020-03-09T00:24:58.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847345+0000", 2020-03-09T00:24:58.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:201 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:24:58.679 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.827753583, 2020-03-09T00:24:58.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784, 2020-03-09T00:24:58.680 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.680 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:58.680 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.680 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.680 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.680 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 201 2020-03-09T00:24:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:24:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:24:58.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081556599999999999 2020-03-09T00:24:58.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838247+0000", 2020-03-09T00:24:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3170000000000004e-06 2020-03-09T00:24:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838255+0000", 2020-03-09T00:24:58.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0073e-05 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838265+0000", 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2199999999999997e-06 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838271+0000", 2020-03-09T00:24:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003565814 2020-03-09T00:24:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841837+0000", 2020-03-09T00:24:58.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013320899999999999 2020-03-09T00:24:58.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841970+0000", 2020-03-09T00:24:58.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001092275 2020-03-09T00:24:58.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843062+0000", 2020-03-09T00:24:58.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003950617 2020-03-09T00:24:58.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847013+0000", 2020-03-09T00:24:58.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067726199999999998 2020-03-09T00:24:58.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:58.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847690+0000", 2020-03-09T00:24:58.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000361431 2020-03-09T00:24:58.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.848052+0000", 2020-03-09T00:24:58.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784 2020-03-09T00:24:58.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.689 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.690 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:202 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:24:58.690 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:24:58.690 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.826866664000001, 2020-03-09T00:24:58.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999, 2020-03-09T00:24:58.691 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.691 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:58.691 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.691 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.691 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.691 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 202 2020-03-09T00:24:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:24:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:24:58.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3191e-05 2020-03-09T00:24:58.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838362+0000", 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2454e-05 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838374+0000", 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8269999999999998e-06 2020-03-09T00:24:58.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838382+0000", 2020-03-09T00:24:58.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1470000000000003e-06 2020-03-09T00:24:58.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838386+0000", 2020-03-09T00:24:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073633309999999999 2020-03-09T00:24:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845750+0000", 2020-03-09T00:24:58.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103528 2020-03-09T00:24:58.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845853+0000", 2020-03-09T00:24:58.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059542799999999995 2020-03-09T00:24:58.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846449+0000", 2020-03-09T00:24:58.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056665000000000003 2020-03-09T00:24:58.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847015+0000", 2020-03-09T00:24:58.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022036690000000001 2020-03-09T00:24:58.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:58.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849219+0000", 2020-03-09T00:24:58.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7259e-05 2020-03-09T00:24:58.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849296+0000", 2020-03-09T00:24:58.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999 2020-03-09T00:24:58.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:203 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:24:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:24:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.826771685000001, 2020-03-09T00:24:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296, 2020-03-09T00:24:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:24:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.702 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 203 2020-03-09T00:24:58.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.703 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:24:58.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4102000000000001e-05 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838438+0000", 2020-03-09T00:24:58.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4809999999999998e-06 2020-03-09T00:24:58.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838441+0000", 2020-03-09T00:24:58.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.691e-06 2020-03-09T00:24:58.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838448+0000", 2020-03-09T00:24:58.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9409999999999999e-06 2020-03-09T00:24:58.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838451+0000", 2020-03-09T00:24:58.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084379850000000003 2020-03-09T00:24:58.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846889+0000", 2020-03-09T00:24:58.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119473 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847008+0000", 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029378160000000002 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849946+0000", 2020-03-09T00:24:58.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129367 2020-03-09T00:24:58.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.850076+0000", 2020-03-09T00:24:58.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013511040000000001 2020-03-09T00:24:58.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.851427+0000", 2020-03-09T00:24:58.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296 2020-03-09T00:24:58.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:24:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:24:58.711 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.823475226999999, 2020-03-09T00:24:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999, 2020-03-09T00:24:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:24:58.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:24:58.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8059000000000002e-05 2020-03-09T00:24:58.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841758+0000", 2020-03-09T00:24:58.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3980000000000005e-06 2020-03-09T00:24:58.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841768+0000", 2020-03-09T00:24:58.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4708e-05 2020-03-09T00:24:58.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841782+0000", 2020-03-09T00:24:58.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1330000000000001e-06 2020-03-09T00:24:58.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841789+0000", 2020-03-09T00:24:58.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037343680000000001 2020-03-09T00:24:58.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845524+0000", 2020-03-09T00:24:58.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9321e-05 2020-03-09T00:24:58.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845543+0000", 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.718 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:24:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:24:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.823319087, 2020-03-09T00:24:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002, 2020-03-09T00:24:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:24:58.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4193e-05 2020-03-09T00:24:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841930+0000", 2020-03-09T00:24:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0671e-05 2020-03-09T00:24:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841941+0000", 2020-03-09T00:24:58.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5038000000000001e-05 2020-03-09T00:24:58.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841956+0000", 2020-03-09T00:24:58.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1e-06 2020-03-09T00:24:58.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841962+0000", 2020-03-09T00:24:58.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036100849999999999 2020-03-09T00:24:58.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845572+0000", 2020-03-09T00:24:58.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117707 2020-03-09T00:24:58.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845690+0000", 2020-03-09T00:24:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002 2020-03-09T00:24:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.726 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:24:58.726 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:24:58.726 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.822813407, 2020-03-09T00:24:58.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002, 2020-03-09T00:24:58.726 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.726 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.726 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:24:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6950000000000002e-05 2020-03-09T00:24:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842429+0000", 2020-03-09T00:24:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1120000000000002e-06 2020-03-09T00:24:58.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842438+0000", 2020-03-09T00:24:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5486999999999999e-05 2020-03-09T00:24:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842453+0000", 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5080000000000002e-06 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842460+0000", 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003166791 2020-03-09T00:24:58.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845627+0000", 2020-03-09T00:24:58.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7489000000000001e-05 2020-03-09T00:24:58.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845644+0000", 2020-03-09T00:24:58.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002 2020-03-09T00:24:58.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.732 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.732 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(osd.3.24:1 3.8s0 3:15387ad6:::smithi10428818-43:head [assert-version v5,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.858252+0000", 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.806933087999999, 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002503211, 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858252+0000", 2020-03-09T00:24:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858252+0000", 2020-03-09T00:24:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9925999999999997e-05 2020-03-09T00:24:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858292+0000", 2020-03-09T00:24:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2719999999999997e-06 2020-03-09T00:24:58.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858297+0000", 2020-03-09T00:24:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0688e-05 2020-03-09T00:24:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858308+0000", 2020-03-09T00:24:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5820000000000004e-06 2020-03-09T00:24:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858316+0000", 2020-03-09T00:24:58.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7987e-05 2020-03-09T00:24:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858364+0000", 2020-03-09T00:24:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7904999999999994e-05 2020-03-09T00:24:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.858442+0000", 2020-03-09T00:24:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002313851 2020-03-09T00:24:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.860755+0000", 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002503211 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 42/32 ECSubWrite(tid=32, reqid=client.4751.0:224, at_version=42'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:24:58.740 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:51.832849+0000", 2020-03-09T00:24:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.8323362840000001, 2020-03-09T00:24:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001720702, 2020-03-09T00:24:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832849+0000", 2020-03-09T00:24:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832849+0000", 2020-03-09T00:24:58.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6902e-05 2020-03-09T00:24:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832886+0000", 2020-03-09T00:24:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6057999999999999e-05 2020-03-09T00:24:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832922+0000", 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3152000000000001e-05 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832945+0000", 2020-03-09T00:24:58.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3510000000000008e-06 2020-03-09T00:24:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.832955+0000", 2020-03-09T00:24:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8042999999999998e-05 2020-03-09T00:24:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.833033+0000", 2020-03-09T00:24:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0514e-05 2020-03-09T00:24:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.833053+0000", 2020-03-09T00:24:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014578530000000001 2020-03-09T00:24:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.834511+0000", 2020-03-09T00:24:58.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8829000000000002e-05 2020-03-09T00:24:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.834570+0000", 2020-03-09T00:24:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001720702 2020-03-09T00:24:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.748 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(osd.12.24:1 3.8s0 3:15387ad6:::smithi10428818-43:head [assert-version v5,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.8130783609999996, 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004, 2020-03-09T00:24:58.749 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.750 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:24:58.750 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:24:58.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:24:58.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0395000000000006e-05 2020-03-09T00:24:58.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852177+0000", 2020-03-09T00:24:58.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1959999999999996e-06 2020-03-09T00:24:58.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852184+0000", 2020-03-09T00:24:58.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0183000000000001e-05 2020-03-09T00:24:58.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852194+0000", 2020-03-09T00:24:58.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.463e-06 2020-03-09T00:24:58.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852199+0000", 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7527e-05 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852257+0000", 2020-03-09T00:24:58.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132466 2020-03-09T00:24:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852389+0000", 2020-03-09T00:24:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042799869999999999 2020-03-09T00:24:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856669+0000", 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:24:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.8021482750000004, 2020-03-09T00:24:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004, 2020-03-09T00:24:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.757 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:24:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022289400000000001 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863260+0000", 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0071000000000001e-05 2020-03-09T00:24:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863280+0000", 2020-03-09T00:24:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2445e-05 2020-03-09T00:24:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863292+0000", 2020-03-09T00:24:58.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2320000000000001e-06 2020-03-09T00:24:58.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863298+0000", 2020-03-09T00:24:58.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7059999999999998e-05 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863365+0000", 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6728e-05 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863381+0000", 2020-03-09T00:24:58.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048764250000000002 2020-03-09T00:24:58.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868258+0000", 2020-03-09T00:24:58.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0857999999999999e-05 2020-03-09T00:24:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868319+0000", 2020-03-09T00:24:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004 2020-03-09T00:24:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.764 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 48/32 ECSubWrite(tid=43, reqid=osd.14.0:43, at_version=48'12, trim_to=0'0, roll_forward_to=44'10))", 2020-03-09T00:24:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:24:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.89257097399999996, 2020-03-09T00:24:58.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001, 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:24:58.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0922999999999999e-05 2020-03-09T00:24:58.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772675+0000", 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8955999999999999e-05 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772704+0000", 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6671e-05 2020-03-09T00:24:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.769 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772721+0000", 2020-03-09T00:24:58.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5750999999999999e-05 2020-03-09T00:24:58.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.769 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772737+0000", 2020-03-09T00:24:58.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0823999999999998e-05 2020-03-09T00:24:58.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.770 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772807+0000", 2020-03-09T00:24:58.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5262000000000001e-05 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772823+0000", 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047643729999999997 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777587+0000", 2020-03-09T00:24:58.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8967999999999993e-05 2020-03-09T00:24:58.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777666+0000", 2020-03-09T00:24:58.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001 2020-03-09T00:24:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.773 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 48/32 ECSubWrite(tid=65, reqid=osd.12.0:65, at_version=48'13, trim_to=0'0, roll_forward_to=44'11))", 2020-03-09T00:24:58.774 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.776465+0000", 2020-03-09T00:24:58.774 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.88871992899999996, 2020-03-09T00:24:58.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019653380000000001, 2020-03-09T00:24:58.774 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.774 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.774 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776465+0000", 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776465+0000", 2020-03-09T00:24:58.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2781999999999995e-05 2020-03-09T00:24:58.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776538+0000", 2020-03-09T00:24:58.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5232000000000002e-05 2020-03-09T00:24:58.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776563+0000", 2020-03-09T00:24:58.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1024000000000001e-05 2020-03-09T00:24:58.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776584+0000", 2020-03-09T00:24:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9849999999999995e-06 2020-03-09T00:24:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776593+0000", 2020-03-09T00:24:58.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3491999999999994e-05 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776667+0000", 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5526e-05 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.779 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.776682+0000", 2020-03-09T00:24:58.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016903899999999999 2020-03-09T00:24:58.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.778373+0000", 2020-03-09T00:24:58.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7907e-05 2020-03-09T00:24:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.778431+0000", 2020-03-09T00:24:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019653380000000001 2020-03-09T00:24:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.781 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.782 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.782 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:275 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~704512] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:24:58.782 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:24:58.782 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.88027924099999999, 2020-03-09T00:24:58.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002, 2020-03-09T00:24:58.783 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.783 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:58.783 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.783 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.783 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.783 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 275 2020-03-09T00:24:58.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:24:58.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013228269999999999 2020-03-09T00:24:58.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786229+0000", 2020-03-09T00:24:58.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9299999999999997e-06 2020-03-09T00:24:58.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786236+0000", 2020-03-09T00:24:58.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3371000000000003e-05 2020-03-09T00:24:58.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786269+0000", 2020-03-09T00:24:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0840000000000003e-06 2020-03-09T00:24:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.787 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786276+0000", 2020-03-09T00:24:58.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7576e-05 2020-03-09T00:24:58.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.788 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786324+0000", 2020-03-09T00:24:58.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010013699999999999 2020-03-09T00:24:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786424+0000", 2020-03-09T00:24:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00139653 2020-03-09T00:24:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787821+0000", 2020-03-09T00:24:58.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089582080000000005 2020-03-09T00:24:58.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796779+0000", 2020-03-09T00:24:58.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036565499999999999 2020-03-09T00:24:58.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:58.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797144+0000", 2020-03-09T00:24:58.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116684 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797261+0000", 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.792 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.793 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:276 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:24:58.793 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:24:58.793 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87794162399999998, 2020-03-09T00:24:58.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001, 2020-03-09T00:24:58.793 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.794 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:58.794 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.794 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.794 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.794 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 276 2020-03-09T00:24:58.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.794 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:24:58.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000487605 2020-03-09T00:24:58.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787731+0000", 2020-03-09T00:24:58.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.875e-06 2020-03-09T00:24:58.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787739+0000", 2020-03-09T00:24:58.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8544000000000002e-05 2020-03-09T00:24:58.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787778+0000", 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4640999999999999e-05 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787792+0000", 2020-03-09T00:24:58.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038656099999999998 2020-03-09T00:24:58.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791658+0000", 2020-03-09T00:24:58.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7368999999999996e-05 2020-03-09T00:24:58.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791745+0000", 2020-03-09T00:24:58.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093415700000000002 2020-03-09T00:24:58.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792680+0000", 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004100289 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796780+0000", 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074439000000000005 2020-03-09T00:24:58.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:58.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797524+0000", 2020-03-09T00:24:58.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2327999999999998e-05 2020-03-09T00:24:58.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797577+0000", 2020-03-09T00:24:58.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001 2020-03-09T00:24:58.803 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.803 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.803 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.804 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:277 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:24:58.804 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:24:58.804 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87729814299999997, 2020-03-09T00:24:58.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006, 2020-03-09T00:24:58.804 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.804 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:24:58.804 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.805 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.805 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.805 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 277 2020-03-09T00:24:58.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.805 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:24:58.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:24:58.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010529100000000001 2020-03-09T00:24:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787992+0000", 2020-03-09T00:24:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.826e-06 2020-03-09T00:24:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787997+0000", 2020-03-09T00:24:58.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0430000000000008e-06 2020-03-09T00:24:58.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788005+0000", 2020-03-09T00:24:58.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1422e-05 2020-03-09T00:24:58.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788017+0000", 2020-03-09T00:24:58.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073510989999999998 2020-03-09T00:24:58.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795368+0000", 2020-03-09T00:24:58.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159238 2020-03-09T00:24:58.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795527+0000", 2020-03-09T00:24:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073327099999999997 2020-03-09T00:24:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:24:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796260+0000", 2020-03-09T00:24:58.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052061200000000005 2020-03-09T00:24:58.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:24:58.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796781+0000", 2020-03-09T00:24:58.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095208999999999995 2020-03-09T00:24:58.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:24:58.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797733+0000", 2020-03-09T00:24:58.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0259000000000001e-05 2020-03-09T00:24:58.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797783+0000", 2020-03-09T00:24:58.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006 2020-03-09T00:24:58.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.814 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.815 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:278 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-03-09T00:24:58.815 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:24:58.815 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87710538599999999, 2020-03-09T00:24:58.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999, 2020-03-09T00:24:58.815 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.815 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:24:58.815 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 278 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:24:58.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:24:58.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3933e-05 2020-03-09T00:24:58.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788134+0000", 2020-03-09T00:24:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2150000000000007e-06 2020-03-09T00:24:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788143+0000", 2020-03-09T00:24:58.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.507e-05 2020-03-09T00:24:58.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788158+0000", 2020-03-09T00:24:58.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.772e-05 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788176+0000", 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010090758 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798267+0000", 2020-03-09T00:24:58.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012688700000000001 2020-03-09T00:24:58.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:24:58.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798394+0000", 2020-03-09T00:24:58.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001284739 2020-03-09T00:24:58.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.799678+0000", 2020-03-09T00:24:58.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999 2020-03-09T00:24:58.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.822 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:24:58.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:24:58.823 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87313220800000002, 2020-03-09T00:24:58.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004, 2020-03-09T00:24:58.824 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.824 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.824 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:24:58.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677000000000002e-05 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792109+0000", 2020-03-09T00:24:58.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1342000000000001e-05 2020-03-09T00:24:58.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792120+0000", 2020-03-09T00:24:58.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8834e-05 2020-03-09T00:24:58.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792139+0000", 2020-03-09T00:24:58.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1979e-05 2020-03-09T00:24:58.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792151+0000", 2020-03-09T00:24:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043798459999999997 2020-03-09T00:24:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796531+0000", 2020-03-09T00:24:58.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.609e-05 2020-03-09T00:24:58.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796547+0000", 2020-03-09T00:24:58.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004 2020-03-09T00:24:58.829 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.830 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:24:58.830 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:24:58.830 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87246590400000001, 2020-03-09T00:24:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002, 2020-03-09T00:24:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:24:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:24:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2404999999999996e-05 2020-03-09T00:24:58.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792782+0000", 2020-03-09T00:24:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2590000000000006e-06 2020-03-09T00:24:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792791+0000", 2020-03-09T00:24:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6633000000000002e-05 2020-03-09T00:24:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792808+0000", 2020-03-09T00:24:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6499999999999999e-06 2020-03-09T00:24:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792814+0000", 2020-03-09T00:24:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003762077 2020-03-09T00:24:58.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796576+0000", 2020-03-09T00:24:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8287e-05 2020-03-09T00:24:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796595+0000", 2020-03-09T00:24:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002 2020-03-09T00:24:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=23, last_complete=48'13, committed=1, applied=1))", 2020-03-09T00:24:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.794867+0000", 2020-03-09T00:24:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.87031813000000002, 2020-03-09T00:24:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017891210000000001, 2020-03-09T00:24:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:24:58.838 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:24:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:24:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:24:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.794867+0000", 2020-03-09T00:24:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:24:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:24:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.794867+0000", 2020-03-09T00:24:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6798999999999998e-05 2020-03-09T00:24:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:24:58.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.794904+0000", 2020-03-09T00:24:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8139999999999997e-06 2020-03-09T00:24:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:24:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.794910+0000", 2020-03-09T00:24:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4800000000000001e-06 2020-03-09T00:24:58.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:24:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.794918+0000", 2020-03-09T00:24:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0590000000000002e-06 2020-03-09T00:24:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:24:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.794923+0000", 2020-03-09T00:24:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017103660000000001 2020-03-09T00:24:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:24:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796634+0000", 2020-03-09T00:24:58.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2603e-05 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796656+0000", 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017891210000000001 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.844 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:24:58.845 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:24:58.845 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:24:58.850 INFO:tasks.rados.rados.0.smithi104.stdout:108: done (2 left) 2020-03-09T00:24:58.850 INFO:tasks.rados.rados.0.smithi104.stdout:112: done (1 left) 2020-03-09T00:24:58.850 INFO:tasks.rados.rados.0.smithi104.stdout:113: read oid 38 snap -1 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:113: expect (ObjNum 37 snap 0 seq_num 37) 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:114: rollback oid 34 current snap is 9 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 34 to 1 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:115: delete oid 33 current snap is 9 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:113: expect (ObjNum 37 snap 0 seq_num 37) 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:114: finishing rollback tid 0 to smithi10428818-34 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:114: finishing rollback tid 1 to smithi10428818-34 2020-03-09T00:24:58.851 INFO:tasks.rados.rados.0.smithi104.stdout:114: finishing rollback tid 2 to smithi10428818-34 2020-03-09T00:24:58.852 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 34 v 25 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-03-09T00:24:58.852 INFO:tasks.rados.rados.0.smithi104.stdout:113: done (3 left) 2020-03-09T00:24:58.852 INFO:tasks.rados.rados.0.smithi104.stdout:114: done (2 left) 2020-03-09T00:24:58.852 INFO:tasks.rados.rados.0.smithi104.stdout:115: done (1 left) 2020-03-09T00:24:58.852 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 42 current snap is 9 2020-03-09T00:24:58.852 INFO:tasks.rados.rados.0.smithi104.stdout:116: seq_num 66 ranges {704512=737280,1441792=376832} 2020-03-09T00:24:58.852 INFO:tasks.rados.rados.0.smithi104.stdout:116: writing smithi10428818-42 from 704512 to 1441792 tid 1 2020-03-09T00:24:58.853 INFO:tasks.rados.rados.0.smithi104.stdout:116: writing smithi10428818-42 from 1441792 to 1818624 tid 2 2020-03-09T00:24:58.853 INFO:tasks.rados.rados.0.smithi104.stdout:117: rollback oid 45 current snap is 9 2020-03-09T00:24:58.853 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 45 to 2 2020-03-09T00:24:58.853 INFO:tasks.rados.rados.0.smithi104.stdout:109: finishing copy_from to smithi10428818-46 2020-03-09T00:24:58.853 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 46 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-03-09T00:24:58.853 INFO:tasks.rados.rados.0.smithi104.stdout:118: setattr oid 40 current snap is 9 2020-03-09T00:24:58.853 INFO:tasks.rados.rados.0.smithi104.stdout:109: finishing copy_from racing read to smithi10428818-46 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:109: done (3 left) 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:119: copy_from oid 22 from oid 35 current snap is 9 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:120: copy_from oid 15 from oid 14 current snap is 9 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 32 current snap is 9 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:121: seq_num 67 ranges {524288=753664,1277952=376832} 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:121: writing smithi10428818-32 from 524288 to 1277952 tid 1 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:121: writing smithi10428818-32 from 1277952 to 1654784 tid 2 2020-03-09T00:24:58.854 INFO:tasks.rados.rados.0.smithi104.stdout:122: rmattr oid 47 current snap is 9 2020-03-09T00:24:58.855 INFO:tasks.rados.rados.0.smithi104.stdout:122: done (6 left) 2020-03-09T00:24:58.855 INFO:tasks.rados.rados.0.smithi104.stdout:123: snap_create 2020-03-09T00:24:58.855 INFO:tasks.rados.rados.0.smithi104.stdout:117: finishing rollback tid 0 to smithi10428818-45 2020-03-09T00:24:58.855 INFO:tasks.rados.rados.0.smithi104.stdout:117: finishing rollback tid 1 to smithi10428818-45 2020-03-09T00:24:58.855 INFO:tasks.rados.rados.0.smithi104.stdout:117: finishing rollback tid 2 to smithi10428818-45 2020-03-09T00:24:58.855 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 45 v 18 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-03-09T00:24:58.855 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 40 v 12 (ObjNum 0 snap 0 seq_num 3966148624) dirty exists 2020-03-09T00:24:58.856 INFO:tasks.rados.rados.0.smithi104.stdout:116: finishing write tid 1 to smithi10428818-42 2020-03-09T00:24:58.856 INFO:tasks.rados.rados.0.smithi104.stdout:116: finishing write tid 2 to smithi10428818-42 2020-03-09T00:24:58.856 INFO:tasks.rados.rados.0.smithi104.stdout:116: finishing write tid 4 to smithi10428818-42 2020-03-09T00:24:58.856 INFO:tasks.rados.rados.0.smithi104.stdout:116: finishing write tid 5 to smithi10428818-42 2020-03-09T00:24:58.856 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 42 v 24 (ObjNum 66 snap 9 seq_num 66) dirty exists 2020-03-09T00:24:58.856 INFO:tasks.rados.rados.0.smithi104.stdout:116: left oid 42 (ObjNum 66 snap 9 seq_num 66) 2020-03-09T00:24:58.856 INFO:tasks.rados.rados.0.smithi104.stdout:120: finishing copy_from to smithi10428818-15 2020-03-09T00:24:58.857 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 15 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-03-09T00:24:58.857 INFO:tasks.rados.rados.0.smithi104.stdout:120: finishing copy_from racing read to smithi10428818-15 2020-03-09T00:24:58.857 INFO:tasks.rados.rados.0.smithi104.stdout:119: finishing copy_from to smithi10428818-22 2020-03-09T00:24:58.857 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 22 v 10 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-03-09T00:24:58.857 INFO:tasks.rados.rados.0.smithi104.stdout:119: finishing copy_from racing read to smithi10428818-22 2020-03-09T00:24:58.857 INFO:tasks.rados.rados.0.smithi104.stdout:121: finishing write tid 1 to smithi10428818-32 2020-03-09T00:24:58.857 INFO:tasks.rados.rados.0.smithi104.stdout:121: finishing write tid 2 to smithi10428818-32 2020-03-09T00:24:58.858 INFO:tasks.rados.rados.0.smithi104.stdout:121: finishing write tid 4 to smithi10428818-32 2020-03-09T00:24:58.858 INFO:tasks.rados.rados.0.smithi104.stdout:121: finishing write tid 5 to smithi10428818-32 2020-03-09T00:24:58.858 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 32 v 14 (ObjNum 67 snap 9 seq_num 67) dirty exists 2020-03-09T00:24:58.858 INFO:tasks.rados.rados.0.smithi104.stdout:121: left oid 32 (ObjNum 67 snap 9 seq_num 67) 2020-03-09T00:24:58.858 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:24:58.843+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:24:58.859 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:24:58.946 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:58.945+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:58.979 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:58.979 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:58.979 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:24:58.980 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:58.988 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:24:59.048 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:59.046+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:59.105 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:59.105 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:24:59.106 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:24:59.106 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:24:59.106 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:59.114 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:24:59.148 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:24:59.147+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:24:59.233 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:24:59.233 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:24:59.233 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:24:59.234 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:24:59.234 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.234 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s3 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:59.234 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.837485+0000", 2020-03-09T00:24:59.234 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.392842065, 2020-03-09T00:24:59.234 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004062937, 2020-03-09T00:24:59.234 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.235 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.235 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.235 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.235 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.235 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837485+0000", 2020-03-09T00:24:59.235 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.236 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.236 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837485+0000", 2020-03-09T00:24:59.236 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030543999999999998 2020-03-09T00:24:59.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.237 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.237 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837791+0000", 2020-03-09T00:24:59.237 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6472999999999999e-05 2020-03-09T00:24:59.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.237 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.237 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.237 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837817+0000", 2020-03-09T00:24:59.238 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4061e-05 2020-03-09T00:24:59.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.238 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.238 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837831+0000", 2020-03-09T00:24:59.238 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4290000000000001e-06 2020-03-09T00:24:59.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.239 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.239 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837840+0000", 2020-03-09T00:24:59.239 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6084999999999997e-05 2020-03-09T00:24:59.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.240 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.240 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837886+0000", 2020-03-09T00:24:59.240 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5550000000000003e-06 2020-03-09T00:24:59.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.240 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.241 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.837894+0000", 2020-03-09T00:24:59.241 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036095260000000001 2020-03-09T00:24:59.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.241 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.241 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.841504+0000", 2020-03-09T00:24:59.242 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4367999999999999e-05 2020-03-09T00:24:59.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.242 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.242 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.242 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.841548+0000", 2020-03-09T00:24:59.242 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004062937 2020-03-09T00:24:59.243 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.243 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.243 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.243 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.243 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.243 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.3s4 38/32 ECSubWrite(tid=36, reqid=client.4751.0:204, at_version=38'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:24:59.244 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.839281+0000", 2020-03-09T00:24:59.244 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.391046907, 2020-03-09T00:24:59.244 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021784980000000001, 2020-03-09T00:24:59.244 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.244 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.244 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.244 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.245 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.245 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839281+0000", 2020-03-09T00:24:59.245 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.245 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.245 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.246 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.246 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839281+0000", 2020-03-09T00:24:59.246 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0448000000000001e-05 2020-03-09T00:24:59.246 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.246 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.246 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.246 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839341+0000", 2020-03-09T00:24:59.247 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9927000000000001e-05 2020-03-09T00:24:59.247 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.247 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.247 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.247 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839361+0000", 2020-03-09T00:24:59.247 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2503999999999999e-05 2020-03-09T00:24:59.248 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.248 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.248 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.248 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839373+0000", 2020-03-09T00:24:59.248 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1369999999999996e-06 2020-03-09T00:24:59.248 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.249 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.249 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.249 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839381+0000", 2020-03-09T00:24:59.249 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3361e-05 2020-03-09T00:24:59.249 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.249 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.250 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.250 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839434+0000", 2020-03-09T00:24:59.250 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4766999999999999e-05 2020-03-09T00:24:59.250 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.250 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.251 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.251 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.839449+0000", 2020-03-09T00:24:59.251 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019603839999999999 2020-03-09T00:24:59.251 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.251 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.841409+0000", 2020-03-09T00:24:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9969999999999998e-05 2020-03-09T00:24:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.841459+0000", 2020-03-09T00:24:59.253 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021784980000000001 2020-03-09T00:24:59.253 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.253 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.253 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.253 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.253 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.254 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s3 38/32 ECSubWrite(tid=16, reqid=client.4751.0:201, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:24:59.254 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.843176+0000", 2020-03-09T00:24:59.254 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.387151738, 2020-03-09T00:24:59.254 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028027960000000002, 2020-03-09T00:24:59.254 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.254 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.255 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.255 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.255 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.255 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843176+0000", 2020-03-09T00:24:59.255 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.255 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.255 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.256 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.256 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843176+0000", 2020-03-09T00:24:59.256 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00037259199999999999 2020-03-09T00:24:59.256 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.256 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.256 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.257 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843548+0000", 2020-03-09T00:24:59.257 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7161999999999999e-05 2020-03-09T00:24:59.257 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.257 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.257 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.257 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843565+0000", 2020-03-09T00:24:59.257 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4263999999999999e-05 2020-03-09T00:24:59.258 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.258 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.258 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.258 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843580+0000", 2020-03-09T00:24:59.258 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8430000000000005e-06 2020-03-09T00:24:59.258 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.259 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.259 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.259 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843590+0000", 2020-03-09T00:24:59.259 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3304999999999999e-05 2020-03-09T00:24:59.259 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.259 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.259 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.260 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843643+0000", 2020-03-09T00:24:59.260 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3050999999999999e-05 2020-03-09T00:24:59.260 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.260 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.260 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.260 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.843656+0000", 2020-03-09T00:24:59.260 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022923169999999999 2020-03-09T00:24:59.261 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.261 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.261 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.261 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.845948+0000", 2020-03-09T00:24:59.261 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0261999999999999e-05 2020-03-09T00:24:59.262 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.262 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.262 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.262 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.845979+0000", 2020-03-09T00:24:59.262 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028027960000000002 2020-03-09T00:24:59.262 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.262 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.263 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.263 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.263 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.263 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s3 44/32 ECSubWrite(tid=39, reqid=client.4751.0:253, at_version=44'10, trim_to=0'0, roll_forward_to=44'8))", 2020-03-09T00:24:59.263 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.851289+0000", 2020-03-09T00:24:59.263 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.3790385919999997, 2020-03-09T00:24:59.264 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002288076, 2020-03-09T00:24:59.264 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.264 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.264 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.264 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.264 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.265 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.851289+0000", 2020-03-09T00:24:59.265 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.265 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.265 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.265 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.265 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.851289+0000", 2020-03-09T00:24:59.266 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3733000000000002e-05 2020-03-09T00:24:59.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.266 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.266 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.266 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.851343+0000", 2020-03-09T00:24:59.266 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1296000000000001e-05 2020-03-09T00:24:59.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.267 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.267 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.851364+0000", 2020-03-09T00:24:59.267 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5392000000000001e-05 2020-03-09T00:24:59.267 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.268 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.268 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.851379+0000", 2020-03-09T00:24:59.268 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5369999999999997e-06 2020-03-09T00:24:59.268 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.268 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.268 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.269 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.851385+0000", 2020-03-09T00:24:59.269 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00080222400000000004 2020-03-09T00:24:59.269 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.269 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.269 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.269 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.852187+0000", 2020-03-09T00:24:59.270 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2670000000000001e-05 2020-03-09T00:24:59.270 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.270 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.270 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.270 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.852200+0000", 2020-03-09T00:24:59.271 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001327476 2020-03-09T00:24:59.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.271 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.271 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.271 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.853527+0000", 2020-03-09T00:24:59.271 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9747999999999999e-05 2020-03-09T00:24:59.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.272 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.272 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.272 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.853577+0000", 2020-03-09T00:24:59.272 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002288076 2020-03-09T00:24:59.272 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.272 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.273 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.273 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.273 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.273 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 44/32 ECSubWrite(tid=54, reqid=client.4751.0:248, at_version=44'21, trim_to=0'0, roll_forward_to=44'17))", 2020-03-09T00:24:59.273 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.855294+0000", 2020-03-09T00:24:59.273 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.3750334970000004, 2020-03-09T00:24:59.274 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030428019999999998, 2020-03-09T00:24:59.274 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.274 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.274 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.274 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.274 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.275 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855294+0000", 2020-03-09T00:24:59.275 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.275 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.275 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.275 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.275 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855294+0000", 2020-03-09T00:24:59.276 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9988000000000002e-05 2020-03-09T00:24:59.276 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.276 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.276 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.276 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855354+0000", 2020-03-09T00:24:59.277 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8203000000000001e-05 2020-03-09T00:24:59.277 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.277 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.277 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.277 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855382+0000", 2020-03-09T00:24:59.277 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6359e-05 2020-03-09T00:24:59.278 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.278 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.278 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.278 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855399+0000", 2020-03-09T00:24:59.279 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0910000000000004e-06 2020-03-09T00:24:59.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.279 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.279 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.279 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855406+0000", 2020-03-09T00:24:59.279 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1760000000000004e-05 2020-03-09T00:24:59.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.280 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.280 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.280 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855477+0000", 2020-03-09T00:24:59.280 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6854000000000002e-05 2020-03-09T00:24:59.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.280 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.281 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.281 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855494+0000", 2020-03-09T00:24:59.281 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027876149999999998 2020-03-09T00:24:59.281 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.281 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.281 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.281 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.858282+0000", 2020-03-09T00:24:59.282 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4932000000000002e-05 2020-03-09T00:24:59.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.282 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.282 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.282 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.858337+0000", 2020-03-09T00:24:59.282 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030428019999999998 2020-03-09T00:24:59.282 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.283 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.283 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.283 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.283 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 44/32 ECSubWrite(tid=55, reqid=client.4751.0:248, at_version=0'0, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:24:59.283 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.862257+0000", 2020-03-09T00:24:59.284 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.3680700459999997, 2020-03-09T00:24:59.284 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034727709999999999, 2020-03-09T00:24:59.284 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.284 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.284 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.284 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.285 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.285 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862257+0000", 2020-03-09T00:24:59.285 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.285 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.285 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.285 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.286 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862257+0000", 2020-03-09T00:24:59.286 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4821000000000002e-05 2020-03-09T00:24:59.286 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.286 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.286 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.286 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862312+0000", 2020-03-09T00:24:59.286 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2026000000000001e-05 2020-03-09T00:24:59.287 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.287 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.287 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.287 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862344+0000", 2020-03-09T00:24:59.287 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6223e-05 2020-03-09T00:24:59.287 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.288 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.288 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.288 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862360+0000", 2020-03-09T00:24:59.288 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7780000000000002e-06 2020-03-09T00:24:59.288 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.288 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.288 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.289 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862367+0000", 2020-03-09T00:24:59.289 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044205999999999998 2020-03-09T00:24:59.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.289 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.289 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.289 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862809+0000", 2020-03-09T00:24:59.289 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3341e-05 2020-03-09T00:24:59.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.290 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.290 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.290 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862823+0000", 2020-03-09T00:24:59.290 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028503920000000002 2020-03-09T00:24:59.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.291 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.291 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.291 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865673+0000", 2020-03-09T00:24:59.291 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7129999999999997e-05 2020-03-09T00:24:59.291 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.291 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.291 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.292 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865730+0000", 2020-03-09T00:24:59.292 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034727709999999999 2020-03-09T00:24:59.292 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.292 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.292 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.292 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.293 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.293 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 44/32 ECSubWrite(tid=56, reqid=client.4751.0:254, at_version=44'23, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:24:59.293 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.863646+0000", 2020-03-09T00:24:59.293 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.3666819370000001, 2020-03-09T00:24:59.293 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002402085, 2020-03-09T00:24:59.293 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.294 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.294 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.294 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.294 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.294 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863646+0000", 2020-03-09T00:24:59.294 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.294 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.295 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.295 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.295 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863646+0000", 2020-03-09T00:24:59.295 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012966599999999999 2020-03-09T00:24:59.295 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.295 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.296 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.296 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863775+0000", 2020-03-09T00:24:59.296 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2280999999999998e-05 2020-03-09T00:24:59.296 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.296 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.296 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.296 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863807+0000", 2020-03-09T00:24:59.297 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0088000000000001e-05 2020-03-09T00:24:59.297 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.297 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.297 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.297 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863828+0000", 2020-03-09T00:24:59.297 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5448000000000001e-05 2020-03-09T00:24:59.298 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.298 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.298 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.298 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863853+0000", 2020-03-09T00:24:59.298 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0541999999999997e-05 2020-03-09T00:24:59.298 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.299 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.299 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863914+0000", 2020-03-09T00:24:59.299 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3267e-05 2020-03-09T00:24:59.299 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.300 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.300 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863927+0000", 2020-03-09T00:24:59.300 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020616060000000001 2020-03-09T00:24:59.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.300 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.301 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.301 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865988+0000", 2020-03-09T00:24:59.301 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9187000000000001e-05 2020-03-09T00:24:59.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.301 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.302 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866048+0000", 2020-03-09T00:24:59.302 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002402085 2020-03-09T00:24:59.302 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.302 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.302 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s3 48/32 ECSubWrite(tid=44, reqid=osd.14.0:43, at_version=0'0, trim_to=0'0, roll_forward_to=48'12))", 2020-03-09T00:24:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:57.779067+0000", 2020-03-09T00:24:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.4512600879999999, 2020-03-09T00:24:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002838124, 2020-03-09T00:24:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.304 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.304 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.304 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779067+0000", 2020-03-09T00:24:59.304 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.304 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.305 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.305 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779067+0000", 2020-03-09T00:24:59.305 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5356999999999999e-05 2020-03-09T00:24:59.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.305 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.305 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.306 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779123+0000", 2020-03-09T00:24:59.306 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.305e-05 2020-03-09T00:24:59.306 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.306 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.306 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.306 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779136+0000", 2020-03-09T00:24:59.307 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4369000000000001e-05 2020-03-09T00:24:59.307 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.307 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.307 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.307 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779150+0000", 2020-03-09T00:24:59.307 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.669e-06 2020-03-09T00:24:59.307 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.308 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.308 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.308 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779157+0000", 2020-03-09T00:24:59.308 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000342216 2020-03-09T00:24:59.308 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.308 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.309 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.309 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779499+0000", 2020-03-09T00:24:59.309 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3751999999999999e-05 2020-03-09T00:24:59.309 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.309 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.309 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.309 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.779523+0000", 2020-03-09T00:24:59.310 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00233154 2020-03-09T00:24:59.310 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.310 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.310 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.310 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.781854+0000", 2020-03-09T00:24:59.310 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1171000000000002e-05 2020-03-09T00:24:59.310 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.311 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.311 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.311 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.781905+0000", 2020-03-09T00:24:59.311 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002838124 2020-03-09T00:24:59.311 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.311 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.311 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.312 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.312 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.312 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s3 48/32 ECSubWrite(tid=22, reqid=client.4751.0:275, at_version=48'12, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:24:59.312 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:57.788459+0000", 2020-03-09T00:24:59.312 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.4418689090000001, 2020-03-09T00:24:59.312 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040104040000000004, 2020-03-09T00:24:59.312 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.313 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.313 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.313 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.313 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.313 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788459+0000", 2020-03-09T00:24:59.313 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788459+0000", 2020-03-09T00:24:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00041410600000000002 2020-03-09T00:24:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.315 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.315 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.315 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788873+0000", 2020-03-09T00:24:59.315 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6750999999999999e-05 2020-03-09T00:24:59.315 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.315 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.315 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.316 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788889+0000", 2020-03-09T00:24:59.316 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5060000000000001e-06 2020-03-09T00:24:59.316 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.316 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.316 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.316 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788899+0000", 2020-03-09T00:24:59.317 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0239999999999997e-06 2020-03-09T00:24:59.317 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.317 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.317 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.317 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788904+0000", 2020-03-09T00:24:59.317 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1972999999999999e-05 2020-03-09T00:24:59.317 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.318 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.318 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.318 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788956+0000", 2020-03-09T00:24:59.318 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2772000000000001e-05 2020-03-09T00:24:59.318 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.318 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.319 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.319 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.788969+0000", 2020-03-09T00:24:59.319 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034641920000000001 2020-03-09T00:24:59.319 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.319 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.319 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.319 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.792433+0000", 2020-03-09T00:24:59.320 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.608e-05 2020-03-09T00:24:59.320 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.320 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.320 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.320 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.792469+0000", 2020-03-09T00:24:59.320 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040104040000000004 2020-03-09T00:24:59.321 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.321 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.321 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.321 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.321 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.321 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s3 48/32 ECSubWrite(tid=23, reqid=client.4751.0:276, at_version=48'13, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:24:59.322 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:57.792708+0000", 2020-03-09T00:24:59.322 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 1.437619403, 2020-03-09T00:24:59.322 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022028939999999999, 2020-03-09T00:24:59.322 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.322 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.322 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.323 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.323 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.323 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.792708+0000", 2020-03-09T00:24:59.323 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.323 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.323 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.324 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.324 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.792708+0000", 2020-03-09T00:24:59.324 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00031796800000000002 2020-03-09T00:24:59.324 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.324 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.324 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.325 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.793026+0000", 2020-03-09T00:24:59.325 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9173e-05 2020-03-09T00:24:59.325 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.325 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.325 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.325 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.793055+0000", 2020-03-09T00:24:59.325 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0988e-05 2020-03-09T00:24:59.326 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.326 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.326 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.326 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.793076+0000", 2020-03-09T00:24:59.326 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1400000000000006e-06 2020-03-09T00:24:59.327 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.327 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.327 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.327 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.793085+0000", 2020-03-09T00:24:59.327 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5268000000000001e-05 2020-03-09T00:24:59.327 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.328 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.328 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.328 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.793131+0000", 2020-03-09T00:24:59.328 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2498e-05 2020-03-09T00:24:59.328 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.328 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.328 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.329 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.793143+0000", 2020-03-09T00:24:59.329 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017323919999999999 2020-03-09T00:24:59.329 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.329 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.329 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.329 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.794875+0000", 2020-03-09T00:24:59.330 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5466999999999998e-05 2020-03-09T00:24:59.330 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.330 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.330 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.330 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.794911+0000", 2020-03-09T00:24:59.330 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022028939999999999 2020-03-09T00:24:59.330 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.331 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.331 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.331 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.331 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.331 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/32 ECSubWrite(tid=71, reqid=client.4751.0:285, at_version=49'23, trim_to=0'0, roll_forward_to=48'21))", 2020-03-09T00:24:59.331 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.776814+0000", 2020-03-09T00:24:59.332 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.45351299900000003, 2020-03-09T00:24:59.332 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002897335, 2020-03-09T00:24:59.332 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.332 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.332 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.332 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.333 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.333 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.776814+0000", 2020-03-09T00:24:59.333 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.333 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.333 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.333 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.333 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.776814+0000", 2020-03-09T00:24:59.334 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0525000000000007e-05 2020-03-09T00:24:59.334 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.334 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.334 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.334 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.776885+0000", 2020-03-09T00:24:59.334 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0584999999999999e-05 2020-03-09T00:24:59.334 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.335 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.335 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.335 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.776916+0000", 2020-03-09T00:24:59.335 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7740999999999998e-05 2020-03-09T00:24:59.335 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.335 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.336 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.336 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.776933+0000", 2020-03-09T00:24:59.336 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1480000000000006e-06 2020-03-09T00:24:59.336 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.336 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.336 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.336 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.776941+0000", 2020-03-09T00:24:59.337 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00075006400000000005 2020-03-09T00:24:59.337 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.337 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.337 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.337 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.777692+0000", 2020-03-09T00:24:59.338 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0381e-05 2020-03-09T00:24:59.338 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.338 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.338 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.338 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.777712+0000", 2020-03-09T00:24:59.338 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019446310000000001 2020-03-09T00:24:59.338 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.339 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.339 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.339 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.779657+0000", 2020-03-09T00:24:59.339 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5260000000000003e-05 2020-03-09T00:24:59.339 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.339 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.340 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.340 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.779712+0000", 2020-03-09T00:24:59.340 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002897335 2020-03-09T00:24:59.340 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.340 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.340 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.340 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.341 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.341 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/32 ECSubWrite(tid=74, reqid=client.4751.0:287, at_version=49'25, trim_to=0'0, roll_forward_to=48'21))", 2020-03-09T00:24:59.341 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.781970+0000", 2020-03-09T00:24:59.341 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.448357122, 2020-03-09T00:24:59.341 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031996260000000001, 2020-03-09T00:24:59.341 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.342 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.342 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.342 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.342 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.342 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.781970+0000", 2020-03-09T00:24:59.342 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.342 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.343 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.343 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.343 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.781970+0000", 2020-03-09T00:24:59.343 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4188000000000003e-05 2020-03-09T00:24:59.343 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.343 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.343 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.344 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782035+0000", 2020-03-09T00:24:59.344 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9089999999999998e-05 2020-03-09T00:24:59.344 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.344 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.344 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.344 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782054+0000", 2020-03-09T00:24:59.345 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4263999999999999e-05 2020-03-09T00:24:59.345 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.345 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.345 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.345 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782068+0000", 2020-03-09T00:24:59.345 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5849999999999999e-06 2020-03-09T00:24:59.346 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.346 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.346 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.346 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782073+0000", 2020-03-09T00:24:59.346 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00052290799999999999 2020-03-09T00:24:59.346 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.346 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.347 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.347 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782596+0000", 2020-03-09T00:24:59.347 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9984e-05 2020-03-09T00:24:59.347 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.347 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.347 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.348 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782616+0000", 2020-03-09T00:24:59.348 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025067850000000001 2020-03-09T00:24:59.348 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.348 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.348 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.348 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.785123+0000", 2020-03-09T00:24:59.348 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6822e-05 2020-03-09T00:24:59.349 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.349 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.349 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.349 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.785170+0000", 2020-03-09T00:24:59.349 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031996260000000001 2020-03-09T00:24:59.349 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.350 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.350 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.350 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.350 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.350 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/32 ECSubWrite(tid=75, reqid=client.4751.0:288, at_version=49'27, trim_to=0'0, roll_forward_to=48'21))", 2020-03-09T00:24:59.350 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.782132+0000", 2020-03-09T00:24:59.351 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.44819511899999998, 2020-03-09T00:24:59.351 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003079287, 2020-03-09T00:24:59.351 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.351 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.351 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.351 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782132+0000", 2020-03-09T00:24:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.352 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782132+0000", 2020-03-09T00:24:59.353 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0585999999999999e-05 2020-03-09T00:24:59.353 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.353 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.353 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.353 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782193+0000", 2020-03-09T00:24:59.353 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3479999999999999e-05 2020-03-09T00:24:59.353 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.354 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.354 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.354 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782216+0000", 2020-03-09T00:24:59.354 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5133999999999999e-05 2020-03-09T00:24:59.354 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.354 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.355 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.355 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782232+0000", 2020-03-09T00:24:59.355 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1149999999999994e-06 2020-03-09T00:24:59.355 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.355 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.355 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.355 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782240+0000", 2020-03-09T00:24:59.356 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00067200300000000001 2020-03-09T00:24:59.356 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.356 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.356 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.356 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782912+0000", 2020-03-09T00:24:59.356 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3190999999999999e-05 2020-03-09T00:24:59.357 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.357 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.357 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.357 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782935+0000", 2020-03-09T00:24:59.357 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022387129999999998 2020-03-09T00:24:59.357 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.358 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.358 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.358 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.785174+0000", 2020-03-09T00:24:59.358 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8065000000000002e-05 2020-03-09T00:24:59.358 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.358 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.359 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.359 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.785212+0000", 2020-03-09T00:24:59.359 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003079287 2020-03-09T00:24:59.359 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.359 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.359 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.359 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.360 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.360 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/48 ECSubWrite(tid=43, reqid=osd.6.0:43, at_version=49'18, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:59.360 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.788670+0000", 2020-03-09T00:24:59.360 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.44165791799999998, 2020-03-09T00:24:59.360 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021047499999999999, 2020-03-09T00:24:59.360 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.361 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.361 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.361 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.361 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.361 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788670+0000", 2020-03-09T00:24:59.361 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.362 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.362 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.362 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.362 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788670+0000", 2020-03-09T00:24:59.362 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3244e-05 2020-03-09T00:24:59.363 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.363 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.363 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.363 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788723+0000", 2020-03-09T00:24:59.363 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5873000000000001e-05 2020-03-09T00:24:59.363 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.363 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.364 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.364 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788749+0000", 2020-03-09T00:24:59.364 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4907e-05 2020-03-09T00:24:59.364 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.364 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.364 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.365 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788764+0000", 2020-03-09T00:24:59.365 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8779999999999998e-06 2020-03-09T00:24:59.365 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.365 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.365 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.365 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788771+0000", 2020-03-09T00:24:59.365 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00043233200000000001 2020-03-09T00:24:59.366 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.366 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.366 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.366 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789204+0000", 2020-03-09T00:24:59.366 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2558000000000001e-05 2020-03-09T00:24:59.366 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.367 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.367 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.367 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789216+0000", 2020-03-09T00:24:59.367 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014974470000000001 2020-03-09T00:24:59.367 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.367 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.367 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.368 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790714+0000", 2020-03-09T00:24:59.368 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0510999999999997e-05 2020-03-09T00:24:59.368 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.368 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.368 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.368 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790774+0000", 2020-03-09T00:24:59.369 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021047499999999999 2020-03-09T00:24:59.369 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.369 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.369 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.369 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.369 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.369 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 49/48 ECSubWrite(tid=69, reqid=osd.11.0:69, at_version=49'25, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:59.370 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.789744+0000", 2020-03-09T00:24:59.370 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.44058364300000002, 2020-03-09T00:24:59.370 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025940429999999999, 2020-03-09T00:24:59.370 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.370 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.370 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.371 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.371 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.371 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789744+0000", 2020-03-09T00:24:59.371 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.371 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.371 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.371 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.372 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789744+0000", 2020-03-09T00:24:59.372 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5202000000000003e-05 2020-03-09T00:24:59.372 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.372 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.372 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.372 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789819+0000", 2020-03-09T00:24:59.372 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7585000000000001e-05 2020-03-09T00:24:59.373 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.373 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.373 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.373 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789847+0000", 2020-03-09T00:24:59.373 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9411000000000001e-05 2020-03-09T00:24:59.373 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.374 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.374 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.374 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789866+0000", 2020-03-09T00:24:59.374 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0060000000000003e-06 2020-03-09T00:24:59.374 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.374 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.375 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.375 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789873+0000", 2020-03-09T00:24:59.375 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109604 2020-03-09T00:24:59.375 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.375 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.375 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.375 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789983+0000", 2020-03-09T00:24:59.376 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1770000000000001e-05 2020-03-09T00:24:59.376 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.376 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.376 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.376 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789994+0000", 2020-03-09T00:24:59.376 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022905379999999999 2020-03-09T00:24:59.376 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.377 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.377 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.377 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.792285+0000", 2020-03-09T00:24:59.377 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2927000000000003e-05 2020-03-09T00:24:59.377 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.377 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.378 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.378 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.792338+0000", 2020-03-09T00:24:59.378 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025940429999999999 2020-03-09T00:24:59.378 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.378 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.379 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.379 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.379 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.379 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 49/48 ECSubWrite(tid=44, reqid=osd.6.0:44, at_version=49'20, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:24:59.379 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.789902+0000", 2020-03-09T00:24:59.379 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.44042583099999999, 2020-03-09T00:24:59.380 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0024474480000000001, 2020-03-09T00:24:59.380 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.380 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.380 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.380 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.380 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.380 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789902+0000", 2020-03-09T00:24:59.381 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.381 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.381 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.381 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.381 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789902+0000", 2020-03-09T00:24:59.381 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4760999999999999e-05 2020-03-09T00:24:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789966+0000", 2020-03-09T00:24:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9278e-05 2020-03-09T00:24:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.382 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.383 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.383 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789996+0000", 2020-03-09T00:24:59.383 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6713000000000001e-05 2020-03-09T00:24:59.383 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.383 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.383 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.383 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790012+0000", 2020-03-09T00:24:59.384 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0280000000000003e-06 2020-03-09T00:24:59.384 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.384 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.384 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.384 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790019+0000", 2020-03-09T00:24:59.384 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000376559 2020-03-09T00:24:59.384 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.385 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.385 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790396+0000", 2020-03-09T00:24:59.385 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2383e-05 2020-03-09T00:24:59.385 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.386 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.386 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790408+0000", 2020-03-09T00:24:59.386 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018900099999999999 2020-03-09T00:24:59.386 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.386 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.386 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.792298+0000", 2020-03-09T00:24:59.387 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0716e-05 2020-03-09T00:24:59.387 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.387 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.387 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.792349+0000", 2020-03-09T00:24:59.387 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0024474480000000001 2020-03-09T00:24:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 49/48 ECSubWrite(tid=76, reqid=client.4751.0:299, at_version=49'30, trim_to=0'0, roll_forward_to=49'29))", 2020-03-09T00:24:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.826787+0000", 2020-03-09T00:24:59.389 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.40354000200000001, 2020-03-09T00:24:59.389 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030971280000000002, 2020-03-09T00:24:59.389 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.389 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.389 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.389 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.390 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.390 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.826787+0000", 2020-03-09T00:24:59.390 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.390 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.390 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.826787+0000", 2020-03-09T00:24:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017116899999999999 2020-03-09T00:24:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.826959+0000", 2020-03-09T00:24:59.391 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6265e-05 2020-03-09T00:24:59.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.392 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.392 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.392 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.826985+0000", 2020-03-09T00:24:59.392 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8947e-05 2020-03-09T00:24:59.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.393 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.393 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827004+0000", 2020-03-09T00:24:59.393 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0139000000000001e-05 2020-03-09T00:24:59.393 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.393 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.394 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827014+0000", 2020-03-09T00:24:59.394 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000101446 2020-03-09T00:24:59.394 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.394 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.394 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.394 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827115+0000", 2020-03-09T00:24:59.395 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3419e-05 2020-03-09T00:24:59.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.395 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.395 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827129+0000", 2020-03-09T00:24:59.395 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002693787 2020-03-09T00:24:59.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.396 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.396 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.396 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829823+0000", 2020-03-09T00:24:59.396 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1956000000000001e-05 2020-03-09T00:24:59.396 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.396 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.397 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.397 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829885+0000", 2020-03-09T00:24:59.397 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030971280000000002 2020-03-09T00:24:59.397 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.397 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.397 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.398 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.398 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.9s3 49/32 ECSubWrite(tid=77, reqid=client.4751.0:297, at_version=49'29, trim_to=0'0, roll_forward_to=49'27))", 2020-03-09T00:24:59.398 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.829020+0000", 2020-03-09T00:24:59.398 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.40130792100000001, 2020-03-09T00:24:59.398 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067514640000000004, 2020-03-09T00:24:59.398 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.399 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.399 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.399 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.399 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.399 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829020+0000", 2020-03-09T00:24:59.399 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.400 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.400 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.400 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.400 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829020+0000", 2020-03-09T00:24:59.400 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016737999999999999 2020-03-09T00:24:59.400 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.400 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.401 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.401 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829187+0000", 2020-03-09T00:24:59.401 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6954999999999999e-05 2020-03-09T00:24:59.401 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.401 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.402 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.402 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829214+0000", 2020-03-09T00:24:59.402 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5183e-05 2020-03-09T00:24:59.402 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.402 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.402 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.403 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829229+0000", 2020-03-09T00:24:59.403 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5010000000000001e-06 2020-03-09T00:24:59.403 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.403 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.403 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.403 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829236+0000", 2020-03-09T00:24:59.404 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9366000000000001e-05 2020-03-09T00:24:59.404 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.404 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.404 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.404 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829285+0000", 2020-03-09T00:24:59.404 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4781e-05 2020-03-09T00:24:59.404 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.405 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.405 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.405 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829300+0000", 2020-03-09T00:24:59.405 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0064100030000000001 2020-03-09T00:24:59.405 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.405 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.405 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.406 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835710+0000", 2020-03-09T00:24:59.406 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1295000000000002e-05 2020-03-09T00:24:59.406 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.406 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.406 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835771+0000", 2020-03-09T00:24:59.407 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067514640000000004 2020-03-09T00:24:59.407 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.407 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.407 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.407 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.407 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.407 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.cs4 49/32 ECSubWrite(tid=77, reqid=client.4751.0:301, at_version=49'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:59.408 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.830703+0000", 2020-03-09T00:24:59.408 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.39962484799999998, 2020-03-09T00:24:59.408 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078474570000000004, 2020-03-09T00:24:59.408 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.408 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.408 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.409 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.409 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.409 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830703+0000", 2020-03-09T00:24:59.409 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.409 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.409 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.410 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.410 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830703+0000", 2020-03-09T00:24:59.410 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00072074599999999997 2020-03-09T00:24:59.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.410 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.410 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.410 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831423+0000", 2020-03-09T00:24:59.411 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6217e-05 2020-03-09T00:24:59.411 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.411 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.411 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.411 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831450+0000", 2020-03-09T00:24:59.411 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9851e-05 2020-03-09T00:24:59.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.412 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.412 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.412 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831469+0000", 2020-03-09T00:24:59.412 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2820000000000002e-06 2020-03-09T00:24:59.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.413 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.413 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.413 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831479+0000", 2020-03-09T00:24:59.413 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2609999999999999e-05 2020-03-09T00:24:59.413 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.413 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.413 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.414 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831531+0000", 2020-03-09T00:24:59.414 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1939999999999998e-06 2020-03-09T00:24:59.414 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.414 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.414 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.414 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831540+0000", 2020-03-09T00:24:59.414 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0069555440000000001 2020-03-09T00:24:59.415 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.415 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.415 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.415 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.838495+0000", 2020-03-09T00:24:59.415 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5013000000000003e-05 2020-03-09T00:24:59.415 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.415 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.416 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.416 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.838550+0000", 2020-03-09T00:24:59.416 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078474570000000004 2020-03-09T00:24:59.416 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.416 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.416 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.417 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.417 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.417 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.cs4 49/32 ECSubWrite(tid=78, reqid=client.4751.0:302, at_version=49'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:24:59.417 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.835238+0000", 2020-03-09T00:24:59.417 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.395089054, 2020-03-09T00:24:59.417 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034908550000000002, 2020-03-09T00:24:59.418 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:24:59.418 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:24:59.418 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:24:59.418 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.418 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:24:59.418 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835238+0000", 2020-03-09T00:24:59.418 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:24:59.419 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.419 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.419 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:24:59.419 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835238+0000", 2020-03-09T00:24:59.419 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049238400000000001 2020-03-09T00:24:59.419 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.420 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:24:59.420 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835731+0000", 2020-03-09T00:24:59.420 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3527999999999999e-05 2020-03-09T00:24:59.420 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.421 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:24:59.421 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835754+0000", 2020-03-09T00:24:59.421 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6877000000000001e-05 2020-03-09T00:24:59.421 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.421 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.421 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:24:59.422 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835771+0000", 2020-03-09T00:24:59.422 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4309999999999997e-06 2020-03-09T00:24:59.422 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.422 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.422 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:24:59.422 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.835779+0000", 2020-03-09T00:24:59.423 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030095099999999999 2020-03-09T00:24:59.423 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.423 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.423 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:24:59.423 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.836080+0000", 2020-03-09T00:24:59.423 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3121e-05 2020-03-09T00:24:59.423 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.424 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:24:59.424 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.836093+0000", 2020-03-09T00:24:59.424 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025877809999999999 2020-03-09T00:24:59.424 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.424 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:24:59.425 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.838680+0000", 2020-03-09T00:24:59.425 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8782000000000002e-05 2020-03-09T00:24:59.425 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:24:59.425 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:24:59.425 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:24:59.425 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.838729+0000", 2020-03-09T00:24:59.426 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034908550000000002 2020-03-09T00:24:59.426 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.426 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.426 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.426 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:24:59.426 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:24:59.427 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:24:59.427 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:24:59.429 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:24:59.427+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:24:59.429 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:24:59.528 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:24:59.527+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:24:59.556 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:59.556 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:59.556 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:24:59.557 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:59.565 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:24:59.601 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:24:59.622 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:24:59.629 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:24:59.628+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:24:59.643 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:24:59.663 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:24:59.681 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:24:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:24:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:24:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:24:59.681 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:24:59.683 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:24:59.689 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:24:59.703 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:24:59.723 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:24:59.732 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:24:59.730+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:24:59.743 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:24:59.763 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:24:59.773 INFO:tasks.rados.rados.0.smithi104.stdout:116: done (6 left) 2020-03-09T00:24:59.773 INFO:tasks.rados.rados.0.smithi104.stdout:117: done (5 left) 2020-03-09T00:24:59.773 INFO:tasks.rados.rados.0.smithi104.stdout:118: done (4 left) 2020-03-09T00:24:59.773 INFO:tasks.rados.rados.0.smithi104.stdout:119: done (3 left) 2020-03-09T00:24:59.774 INFO:tasks.rados.rados.0.smithi104.stdout:120: done (2 left) 2020-03-09T00:24:59.774 INFO:tasks.rados.rados.0.smithi104.stdout:121: done (1 left) 2020-03-09T00:24:59.774 INFO:tasks.rados.rados.0.smithi104.stdout:123: done (0 left) 2020-03-09T00:24:59.774 INFO:tasks.rados.rados.0.smithi104.stdout:124: snap_create 2020-03-09T00:24:59.786 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:24:59.798 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-03-09T00:24:59.805 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:59.805 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:59.807 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:24:59.814 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-03-09T00:24:59.829 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:24:59.831 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:24:59.829+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:24:59.849 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:24:59.850 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:24:59.848+0000 7f51feffd700 0 -- 172.21.15.148:0/1984385568 >> [v2:172.21.15.50:6816/27631,v1:172.21.15.50:6817/27631] conn(0x7f51e402c0d0 msgr2=0x7f51e402e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:24:59.871 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15] out_osds: [13] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:24:59.871 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:24:59.872 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15] 2020-03-09T00:24:59.872 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd out 5 2020-03-09T00:24:59.931 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:24:59.930+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:24:59.941 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:24:59.942 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:24:59.950 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-03-09T00:25:00.032 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:00.031+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:00.072 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:00.092 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:00.107 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:00.108 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:00.113 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:00.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.116 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.117 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:00.118 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-03-09T00:25:00.133 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:00.132+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:00.158 INFO:tasks.ceph.mon.b.smithi050.stderr:2020-03-09T00:25:00.156+0000 7fa09607d700 -1 mon.b@0(leader).osd e50 definitely_dead 0 2020-03-09T00:25:00.233 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:00.234 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:00.234 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:00.232+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:00.242 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:00.335 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:00.334+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:00.378 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:00.378 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:00.390 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:00.438 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:00.436+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:00.514 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:00.515 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:00.523 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:00.539 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:00.538+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:00.640 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:00.638+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:00.648 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:00.648 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:00.656 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:25:00.741 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:00.740+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:00.775 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:00.775 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:00.778 INFO:tasks.rados.rados.0.smithi104.stdout:124: done (0 left) 2020-03-09T00:25:00.778 INFO:tasks.rados.rados.0.smithi104.stdout:125: read oid 9 snap -1 2020-03-09T00:25:00.778 INFO:tasks.rados.rados.0.smithi104.stdout:125: expect deleted 2020-03-09T00:25:00.778 INFO:tasks.rados.rados.0.smithi104.stdout:126: snap_remove snap 1 2020-03-09T00:25:00.783 INFO:teuthology.orchestra.run.smithi148.stderr:marked out osd.5. 2020-03-09T00:25:00.786 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is unset 2020-03-09T00:25:00.790 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:25:00.843 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:00.841+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:00.906 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:00.906 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:00.914 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:25:00.944 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:00.943+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:01.030 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:01.030 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:01.038 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:25:01.082 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:01.081+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:01.145 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:01.144+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:01.161 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:01.161 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:01.169 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:25:01.246 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:01.245+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:01.324 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:01.324 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:01.332 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:25:01.375 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:01.374+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:01.447 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:01.447 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:01.448 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:01.446+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:01.455 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:25:01.548 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:01.547+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:01.582 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:01.582 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:01.591 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:25:01.649 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:01.647+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:01.747 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:01.748 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:01.750 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:01.748+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:01.756 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:25:01.762 INFO:tasks.rados.rados.0.smithi104.stdout:126: done (1 left) 2020-03-09T00:25:01.762 INFO:tasks.rados.rados.0.smithi104.stdout:127: rmattr oid 29 current snap is 11 2020-03-09T00:25:01.762 INFO:tasks.rados.rados.0.smithi104.stdout:127: done (1 left) 2020-03-09T00:25:01.762 INFO:tasks.rados.rados.0.smithi104.stdout:128: rmattr oid 44 current snap is 11 2020-03-09T00:25:01.762 INFO:tasks.rados.rados.0.smithi104.stdout:128: done (1 left) 2020-03-09T00:25:01.763 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 23 current snap is 11 2020-03-09T00:25:01.763 INFO:tasks.rados.rados.0.smithi104.stdout:129: seq_num 68 ranges {0=770048,770048=376832} 2020-03-09T00:25:01.767 INFO:tasks.rados.rados.0.smithi104.stdout:129: writing smithi10428818-23 from 0 to 770048 tid 1 2020-03-09T00:25:01.770 INFO:tasks.rados.rados.0.smithi104.stdout:129: writing smithi10428818-23 from 770048 to 1146880 tid 2 2020-03-09T00:25:01.770 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 25 current snap is 11 2020-03-09T00:25:01.770 INFO:tasks.rados.rados.0.smithi104.stdout:130: seq_num 69 ranges {409600=786432,1196032=376832} 2020-03-09T00:25:01.782 INFO:tasks.rados.rados.0.smithi104.stdout:130: writing smithi10428818-25 from 409600 to 1196032 tid 1 2020-03-09T00:25:01.785 INFO:tasks.rados.rados.0.smithi104.stdout:130: writing smithi10428818-25 from 1196032 to 1572864 tid 2 2020-03-09T00:25:01.785 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 16 current snap is 11 2020-03-09T00:25:01.785 INFO:tasks.rados.rados.0.smithi104.stdout:131: seq_num 70 ranges {262144=802816,1064960=376832} 2020-03-09T00:25:01.791 INFO:tasks.rados.rados.0.smithi104.stdout:131: writing smithi10428818-16 from 262144 to 1064960 tid 1 2020-03-09T00:25:01.794 INFO:tasks.rados.rados.0.smithi104.stdout:131: writing smithi10428818-16 from 1064960 to 1441792 tid 2 2020-03-09T00:25:01.794 INFO:tasks.rados.rados.0.smithi104.stdout:132: rollback oid 49 current snap is 11 2020-03-09T00:25:01.795 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 49 to 9 2020-03-09T00:25:01.795 INFO:tasks.rados.rados.0.smithi104.stdout:133: copy_from oid 13 from oid 21 current snap is 11 2020-03-09T00:25:01.795 INFO:tasks.rados.rados.0.smithi104.stdout:134: copy_from oid 4 from oid 22 current snap is 11 2020-03-09T00:25:01.795 INFO:tasks.rados.rados.0.smithi104.stdout:135: copy_from oid 50 from oid 47 current snap is 11 2020-03-09T00:25:01.795 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 42 current snap is 11 2020-03-09T00:25:01.795 INFO:tasks.rados.rados.0.smithi104.stdout:136: seq_num 71 ranges {1818624=425984,2244608=573440,2818048=196608} 2020-03-09T00:25:01.806 INFO:tasks.rados.rados.0.smithi104.stdout:136: writing smithi10428818-42 from 1818624 to 2244608 tid 1 2020-03-09T00:25:01.810 INFO:tasks.rados.rados.0.smithi104.stdout:136: writing smithi10428818-42 from 2244608 to 2818048 tid 2 2020-03-09T00:25:01.811 INFO:tasks.rados.rados.0.smithi104.stdout:136: writing smithi10428818-42 from 2818048 to 3014656 tid 3 2020-03-09T00:25:01.812 INFO:tasks.rados.rados.0.smithi104.stdout:137: delete oid 3 current snap is 11 2020-03-09T00:25:01.812 INFO:tasks.rados.rados.0.smithi104.stdout:130: finishing write tid 1 to smithi10428818-25 2020-03-09T00:25:01.812 INFO:tasks.rados.rados.0.smithi104.stdout:130: finishing write tid 2 to smithi10428818-25 2020-03-09T00:25:01.812 INFO:tasks.rados.rados.0.smithi104.stdout:133: finishing copy_from to smithi10428818-13 2020-03-09T00:25:01.812 INFO:tasks.rados.rados.0.smithi104.stdout:133: got expected ENOENT (src dne) 2020-03-09T00:25:01.812 INFO:tasks.rados.rados.0.smithi104.stdout:133: finishing copy_from racing read to smithi10428818-13 2020-03-09T00:25:01.812 INFO:tasks.rados.rados.0.smithi104.stdout:130: finishing write tid 4 to smithi10428818-25 2020-03-09T00:25:01.813 INFO:tasks.rados.rados.0.smithi104.stdout:130: finishing write tid 5 to smithi10428818-25 2020-03-09T00:25:01.813 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 25 v 14 (ObjNum 69 snap 11 seq_num 69) dirty exists 2020-03-09T00:25:01.813 INFO:tasks.rados.rados.0.smithi104.stdout:130: left oid 25 (ObjNum 69 snap 11 seq_num 69) 2020-03-09T00:25:01.813 INFO:tasks.rados.rados.0.smithi104.stdout:131: finishing write tid 1 to smithi10428818-16 2020-03-09T00:25:01.813 INFO:tasks.rados.rados.0.smithi104.stdout:131: finishing write tid 2 to smithi10428818-16 2020-03-09T00:25:01.813 INFO:tasks.rados.rados.0.smithi104.stdout:131: finishing write tid 4 to smithi10428818-16 2020-03-09T00:25:01.813 INFO:tasks.rados.rados.0.smithi104.stdout:131: finishing write tid 5 to smithi10428818-16 2020-03-09T00:25:01.814 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 16 v 41 (ObjNum 70 snap 11 seq_num 70) dirty exists 2020-03-09T00:25:01.814 INFO:tasks.rados.rados.0.smithi104.stdout:131: left oid 16 (ObjNum 70 snap 11 seq_num 70) 2020-03-09T00:25:01.818 INFO:tasks.rados.rados.0.smithi104.stdout:136: finishing write tid 1 to smithi10428818-42 2020-03-09T00:25:01.819 INFO:tasks.rados.rados.0.smithi104.stdout:136: finishing write tid 2 to smithi10428818-42 2020-03-09T00:25:01.819 INFO:tasks.rados.rados.0.smithi104.stdout:136: finishing write tid 3 to smithi10428818-42 2020-03-09T00:25:01.821 INFO:tasks.rados.rados.0.smithi104.stdout:136: finishing write tid 5 to smithi10428818-42 2020-03-09T00:25:01.823 INFO:tasks.rados.rados.0.smithi104.stdout:136: finishing write tid 6 to smithi10428818-42 2020-03-09T00:25:01.823 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 42 v 35 (ObjNum 71 snap 11 seq_num 71) dirty exists 2020-03-09T00:25:01.823 INFO:tasks.rados.rados.0.smithi104.stdout:136: left oid 42 (ObjNum 71 snap 11 seq_num 71) 2020-03-09T00:25:01.850 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:01.849+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:01.916 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:01.916 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:01.924 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:25:01.951 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:01.950+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:02.052 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:02.050+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:02.056 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:02.057 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:02.065 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:25:02.124 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:02.152 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:02.151+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:02.229 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:02.229 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:02.238 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:25:02.258 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:02.278 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:02.277+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:02.279 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:02.299 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:02.319 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:02.338 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:02.354 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:02.353+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:02.358 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:02.378 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:25:02.398 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:02.400 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:02.401 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:02.409 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:25:02.418 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:02.440 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:02.455 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:02.454+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:02.460 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:02.481 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:02.501 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:02.522 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:02.522+0000 7fe0b659c700 0 -- 172.21.15.148:0/481925423 >> [v2:172.21.15.139:6818/26981,v1:172.21.15.139:6821/26981] conn(0x7fe09c02f3e0 msgr2=0x7fe09c0317d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:02.532 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:02.533 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:02.533 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:02.533 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:02.544 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:25:02.553 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:02.556 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:02.555+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:02.574 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:02.575 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.575 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.575 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.575 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.575 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.576 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.576 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.576 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.576 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.576 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.576 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.576 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.577 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.577 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.577 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.577 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:02.658 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:02.656+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:02.668 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:02.669 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:02.669 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:02.669 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:02.669 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:02.677 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:25:02.759 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:02.758+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:02.793 INFO:tasks.rados.rados.0.smithi104.stdout:132: finishing rollback tid 0 to smithi10428818-49 2020-03-09T00:25:02.794 INFO:tasks.rados.rados.0.smithi104.stdout:125: done (9 left) 2020-03-09T00:25:02.795 INFO:tasks.rados.rados.0.smithi104.stdout:130: done (8 left) 2020-03-09T00:25:02.795 INFO:tasks.rados.rados.0.smithi104.stdout:131: done (7 left) 2020-03-09T00:25:02.795 INFO:tasks.rados.rados.0.smithi104.stdout:133: done (6 left) 2020-03-09T00:25:02.795 INFO:tasks.rados.rados.0.smithi104.stdout:136: done (5 left) 2020-03-09T00:25:02.795 INFO:tasks.rados.rados.0.smithi104.stdout:137: done (4 left) 2020-03-09T00:25:02.795 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 20 current snap is 11 2020-03-09T00:25:02.795 INFO:tasks.rados.rados.0.smithi104.stdout:138: seq_num 73 ranges {1343488=458752,1802240=475136,2277376=294912} 2020-03-09T00:25:02.799 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-03-09T00:25:02.805 INFO:tasks.rados.rados.0.smithi104.stdout:138: writing smithi10428818-20 from 1343488 to 1802240 tid 1 2020-03-09T00:25:02.807 INFO:tasks.rados.rados.0.smithi104.stdout:138: writing smithi10428818-20 from 1802240 to 2277376 tid 2 2020-03-09T00:25:02.810 INFO:tasks.rados.rados.0.smithi104.stdout:138: writing smithi10428818-20 from 2277376 to 2572288 tid 3 2020-03-09T00:25:02.810 INFO:tasks.rados.rados.0.smithi104.stdout:139: setattr oid 16 current snap is 11 2020-03-09T00:25:02.810 INFO:tasks.rados.rados.0.smithi104.stdout:129: finishing write tid 1 to smithi10428818-23 2020-03-09T00:25:02.810 INFO:tasks.rados.rados.0.smithi104.stdout:129: finishing write tid 2 to smithi10428818-23 2020-03-09T00:25:02.810 INFO:tasks.rados.rados.0.smithi104.stdout:129: finishing write tid 4 to smithi10428818-23 2020-03-09T00:25:02.811 INFO:tasks.rados.rados.0.smithi104.stdout:135: finishing copy_from to smithi10428818-50 2020-03-09T00:25:02.811 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 50 v 10 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-03-09T00:25:02.811 INFO:tasks.rados.rados.0.smithi104.stdout:135: finishing copy_from racing read to smithi10428818-50 2020-03-09T00:25:02.811 INFO:tasks.rados.rados.0.smithi104.stdout:129: finishing write tid 5 to smithi10428818-23 2020-03-09T00:25:02.811 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 23 v 15 (ObjNum 68 snap 11 seq_num 68) dirty exists 2020-03-09T00:25:02.811 INFO:tasks.rados.rados.0.smithi104.stdout:129: left oid 23 (ObjNum 68 snap 11 seq_num 68) 2020-03-09T00:25:02.811 INFO:tasks.rados.rados.0.smithi104.stdout:134: finishing copy_from to smithi10428818-4 2020-03-09T00:25:02.812 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 4 v 10 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-03-09T00:25:02.813 INFO:tasks.rados.rados.0.smithi104.stdout:134: finishing copy_from racing read to smithi10428818-4 2020-03-09T00:25:02.813 INFO:tasks.rados.rados.0.smithi104.stdout:129: done (5 left) 2020-03-09T00:25:02.813 INFO:tasks.rados.rados.0.smithi104.stdout:134: done (4 left) 2020-03-09T00:25:02.813 INFO:tasks.rados.rados.0.smithi104.stdout:135: done (3 left) 2020-03-09T00:25:02.813 INFO:tasks.rados.rados.0.smithi104.stdout:140: copy_from oid 48 from oid 42 current snap is 11 2020-03-09T00:25:02.813 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 8 current snap is 11 2020-03-09T00:25:02.814 INFO:tasks.rados.rados.0.smithi104.stdout:141: seq_num 74 ranges {1998848=475136,2473984=622592,3096576=147456} 2020-03-09T00:25:02.826 INFO:tasks.rados.rados.0.smithi104.stdout:141: writing smithi10428818-8 from 1998848 to 2473984 tid 1 2020-03-09T00:25:02.830 INFO:tasks.rados.rados.0.smithi104.stdout:141: writing smithi10428818-8 from 2473984 to 3096576 tid 2 2020-03-09T00:25:02.831 INFO:tasks.rados.rados.0.smithi104.stdout:141: writing smithi10428818-8 from 3096576 to 3244032 tid 3 2020-03-09T00:25:02.831 INFO:tasks.rados.rados.0.smithi104.stdout:142: read oid 10 snap -1 2020-03-09T00:25:02.831 INFO:tasks.rados.rados.0.smithi104.stdout:142: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:25:02.832 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 12 current snap is 11 2020-03-09T00:25:02.832 INFO:tasks.rados.rados.0.smithi104.stdout:143: seq_num 75 ranges {1261568=491520,1753088=770048} 2020-03-09T00:25:02.841 INFO:tasks.rados.rados.0.smithi104.stdout:143: writing smithi10428818-12 from 1261568 to 1753088 tid 1 2020-03-09T00:25:02.847 INFO:tasks.rados.rados.0.smithi104.stdout:143: writing smithi10428818-12 from 1753088 to 2523136 tid 2 2020-03-09T00:25:02.847 INFO:tasks.rados.rados.0.smithi104.stdout:144: setattr oid 44 current snap is 11 2020-03-09T00:25:02.849 INFO:tasks.rados.rados.0.smithi104.stdout:138: finishing write tid 1 to smithi10428818-20 2020-03-09T00:25:02.849 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 16 v 42 (ObjNum 70 snap 11 seq_num 70) dirty exists 2020-03-09T00:25:02.850 INFO:tasks.rados.rados.0.smithi104.stdout:138: finishing write tid 2 to smithi10428818-20 2020-03-09T00:25:02.850 INFO:tasks.rados.rados.0.smithi104.stdout:138: finishing write tid 3 to smithi10428818-20 2020-03-09T00:25:02.850 INFO:tasks.rados.rados.0.smithi104.stdout:138: finishing write tid 5 to smithi10428818-20 2020-03-09T00:25:02.850 INFO:tasks.rados.rados.0.smithi104.stdout:138: finishing write tid 6 to smithi10428818-20 2020-03-09T00:25:02.850 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 20 v 40 (ObjNum 73 snap 11 seq_num 73) dirty exists 2020-03-09T00:25:02.850 INFO:tasks.rados.rados.0.smithi104.stdout:138: left oid 20 (ObjNum 73 snap 11 seq_num 73) 2020-03-09T00:25:02.851 INFO:tasks.rados.rados.0.smithi104.stdout:141: finishing write tid 1 to smithi10428818-8 2020-03-09T00:25:02.851 INFO:tasks.rados.rados.0.smithi104.stdout:142: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:25:02.860 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:02.858+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:02.883 INFO:tasks.rados.rados.0.smithi104.stdout:141: finishing write tid 2 to smithi10428818-8 2020-03-09T00:25:02.883 INFO:tasks.rados.rados.0.smithi104.stdout:141: finishing write tid 3 to smithi10428818-8 2020-03-09T00:25:02.883 INFO:tasks.rados.rados.0.smithi104.stdout:141: finishing write tid 5 to smithi10428818-8 2020-03-09T00:25:02.883 INFO:tasks.rados.rados.0.smithi104.stdout:141: finishing write tid 6 to smithi10428818-8 2020-03-09T00:25:02.884 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 8 v 21 (ObjNum 74 snap 11 seq_num 74) dirty exists 2020-03-09T00:25:02.884 INFO:tasks.rados.rados.0.smithi104.stdout:141: left oid 8 (ObjNum 74 snap 11 seq_num 74) 2020-03-09T00:25:02.884 INFO:tasks.rados.rados.0.smithi104.stdout:143: finishing write tid 1 to smithi10428818-12 2020-03-09T00:25:02.884 INFO:tasks.rados.rados.0.smithi104.stdout:143: finishing write tid 2 to smithi10428818-12 2020-03-09T00:25:02.884 INFO:tasks.rados.rados.0.smithi104.stdout:143: finishing write tid 4 to smithi10428818-12 2020-03-09T00:25:02.884 INFO:tasks.rados.rados.0.smithi104.stdout:143: finishing write tid 5 to smithi10428818-12 2020-03-09T00:25:02.884 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 12 v 24 (ObjNum 75 snap 11 seq_num 75) dirty exists 2020-03-09T00:25:02.885 INFO:tasks.rados.rados.0.smithi104.stdout:143: left oid 12 (ObjNum 75 snap 11 seq_num 75) 2020-03-09T00:25:02.885 INFO:tasks.rados.rados.0.smithi104.stdout:140: finishing copy_from to smithi10428818-48 2020-03-09T00:25:02.885 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 48 v 35 (ObjNum 71 snap 11 seq_num 71) dirty exists 2020-03-09T00:25:02.885 INFO:tasks.rados.rados.0.smithi104.stdout:140: finishing copy_from racing read to smithi10428818-48 2020-03-09T00:25:02.885 INFO:tasks.rados.rados.0.smithi104.stdout:138: done (7 left) 2020-03-09T00:25:02.885 INFO:tasks.rados.rados.0.smithi104.stdout:139: done (6 left) 2020-03-09T00:25:02.885 INFO:tasks.rados.rados.0.smithi104.stdout:140: done (5 left) 2020-03-09T00:25:02.886 INFO:tasks.rados.rados.0.smithi104.stdout:141: done (4 left) 2020-03-09T00:25:02.886 INFO:tasks.rados.rados.0.smithi104.stdout:142: done (3 left) 2020-03-09T00:25:02.886 INFO:tasks.rados.rados.0.smithi104.stdout:143: done (2 left) 2020-03-09T00:25:02.886 INFO:tasks.rados.rados.0.smithi104.stdout:145: read oid 17 snap -1 2020-03-09T00:25:02.886 INFO:tasks.rados.rados.0.smithi104.stdout:145: expect (ObjNum 31 snap 0 seq_num 31) 2020-03-09T00:25:02.886 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 18 current snap is 11 2020-03-09T00:25:02.887 INFO:tasks.rados.rados.0.smithi104.stdout:146: seq_num 76 ranges {1294336=507904,1802240=507904,2310144=262144} 2020-03-09T00:25:02.888 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:02.888 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:02.889 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:02.889 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:02.889 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.889 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:195 3.9s0 3:93e8184b:::smithi10428818-7:head [stat,delete] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:02.890 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.811698+0000", 2020-03-09T00:25:02.890 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.068374631999999, 2020-03-09T00:25:02.890 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097068750000000002, 2020-03-09T00:25:02.890 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.890 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:02.890 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.890 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.891 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.891 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 195 2020-03-09T00:25:02.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.891 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811698+0000", 2020-03-09T00:25:02.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811698+0000", 2020-03-09T00:25:02.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6551000000000002e-05 2020-03-09T00:25:02.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.893 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811745+0000", 2020-03-09T00:25:02.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2041e-05 2020-03-09T00:25:02.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.894 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811757+0000", 2020-03-09T00:25:02.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6067999999999999e-05 2020-03-09T00:25:02.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811773+0000", 2020-03-09T00:25:02.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9880000000000004e-06 2020-03-09T00:25:02.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.896 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811781+0000", 2020-03-09T00:25:02.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7058999999999996e-05 2020-03-09T00:25:02.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.896 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811848+0000", 2020-03-09T00:25:02.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2191999999999999e-05 2020-03-09T00:25:02.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.897 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:02.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811860+0000", 2020-03-09T00:25:02.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0024088339999999999 2020-03-09T00:25:02.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814269+0000", 2020-03-09T00:25:02.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9890999999999996e-05 2020-03-09T00:25:02.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814369+0000", 2020-03-09T00:25:02.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00098004199999999998 2020-03-09T00:25:02.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:02.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815349+0000", 2020-03-09T00:25:02.900 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011032940000000001 2020-03-09T00:25:02.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:02.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.816452+0000", 2020-03-09T00:25:02.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048642119999999997 2020-03-09T00:25:02.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:02.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.821317+0000", 2020-03-09T00:25:02.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8702999999999994e-05 2020-03-09T00:25:02.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.821405+0000", 2020-03-09T00:25:02.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097068750000000002 2020-03-09T00:25:02.902 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.902 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.903 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.903 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:225 3.0s0 3:054f30c0:::smithi10428818-49:head [read 0~819200,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:25:02.903 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:51.848356+0000", 2020-03-09T00:25:02.903 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.031717206, 2020-03-09T00:25:02.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013427335, 2020-03-09T00:25:02.904 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.904 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:02.904 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.904 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.904 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.904 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 225 2020-03-09T00:25:02.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.905 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848356+0000", 2020-03-09T00:25:02.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848356+0000", 2020-03-09T00:25:02.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2160000000000003e-05 2020-03-09T00:25:02.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848398+0000", 2020-03-09T00:25:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1635e-05 2020-03-09T00:25:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848410+0000", 2020-03-09T00:25:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2568999999999999e-05 2020-03-09T00:25:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848422+0000", 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0365e-05 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848433+0000", 2020-03-09T00:25:02.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8063999999999998e-05 2020-03-09T00:25:02.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.909 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.909 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848491+0000", 2020-03-09T00:25:02.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8150000000000006e-06 2020-03-09T00:25:02.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.910 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:02.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848498+0000", 2020-03-09T00:25:02.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021196790000000002 2020-03-09T00:25:02.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.850618+0000", 2020-03-09T00:25:02.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4110999999999996e-05 2020-03-09T00:25:02.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.850692+0000", 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011090937 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.861783+0000", 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013427335 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.912 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.913 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.913 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:226 3.0s0 3:054f30c0:::smithi10428818-49:head [read 0~819200,checksum] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:25:02.913 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:51.848471+0000", 2020-03-09T00:25:02.913 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.031602266, 2020-03-09T00:25:02.913 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014676102999999999, 2020-03-09T00:25:02.914 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.914 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:02.914 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.914 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.914 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.915 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 226 2020-03-09T00:25:02.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.915 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848471+0000", 2020-03-09T00:25:02.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848471+0000", 2020-03-09T00:25:02.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7950000000000001e-05 2020-03-09T00:25:02.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848509+0000", 2020-03-09T00:25:02.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9469999999999996e-06 2020-03-09T00:25:02.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848513+0000", 2020-03-09T00:25:02.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4509999999999997e-06 2020-03-09T00:25:02.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848519+0000", 2020-03-09T00:25:02.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.179e-06 2020-03-09T00:25:02.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.919 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.919 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848522+0000", 2020-03-09T00:25:02.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9040999999999999e-05 2020-03-09T00:25:02.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.920 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.920 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848551+0000", 2020-03-09T00:25:02.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5090000000000004e-06 2020-03-09T00:25:02.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.921 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:02.921 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848556+0000", 2020-03-09T00:25:02.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0026038540000000001 2020-03-09T00:25:02.921 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.921 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.851160+0000", 2020-03-09T00:25:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000116722 2020-03-09T00:25:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.851276+0000", 2020-03-09T00:25:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011870449999999999 2020-03-09T00:25:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.863147+0000", 2020-03-09T00:25:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014676102999999999 2020-03-09T00:25:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:227 3.0s0 3:054f30c0:::smithi10428818-49:head [read 0~819200,checksum] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:25:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:51.848563+0000", 2020-03-09T00:25:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.031510279999999, 2020-03-09T00:25:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015311103, 2020-03-09T00:25:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 227 2020-03-09T00:25:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848563+0000", 2020-03-09T00:25:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848563+0000", 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3482999999999997e-05 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848606+0000", 2020-03-09T00:25:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0260000000000001e-06 2020-03-09T00:25:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848610+0000", 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2890000000000003e-06 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848617+0000", 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8760000000000001e-06 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848619+0000", 2020-03-09T00:25:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7015999999999999e-05 2020-03-09T00:25:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848656+0000", 2020-03-09T00:25:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.212e-06 2020-03-09T00:25:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848663+0000", 2020-03-09T00:25:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0031832269999999998 2020-03-09T00:25:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.851846+0000", 2020-03-09T00:25:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010709600000000001 2020-03-09T00:25:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.851953+0000", 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011920877999999999 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.863874+0000", 2020-03-09T00:25:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015311103 2020-03-09T00:25:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:228 3.9s0 3:93e8184b:::smithi10428818-7:head [create,setxattr rYWl][jZaQdY`hOflfsigtpQplO]_[olsUp[Ziak (7228) in=40b,setxattr SY[oXm_PUVnSaU\\giaqa^\\YRdSeahaS\\p_TkO[UU (103) in=40b,setxattr l_b`Ra]^NacYVTZ\\^c]OWU\\tN`rRZt`S[TVOmg^e (559) in=40b,setxattr t`hmjchn[UiYidtcbbp`eaUfqnqTNRk\\ci^amom` (1649) in=40b,setxattr VnrjXe`XfobqpncPaY_ftpYr__]hqq^s[qgYriOc (771) in=40b,setxattr YnmkmVsjfO^idcWk_WYmU`cpdUjgaXVZdXjlnlSO (1051) in=40b,setxattr pjsehh_lsVdmqljOXnhjbVrgbaUta` (806) in=30b,setxattr OOloQZVedk[_SPZf[PtYmZqWikYiiobZoYeRiaUl (3368) in=40b,setxattr Soa\\ri[fdN[Sfba_j (806) in=17b,setxattr kQYgOfs\\kjTagmjZhTrl`boXreb^Yfnjts\\rShmQ (1998) in=40b,setxattr qlTNiOdhXV]^`ibS`]^QZtPOOZriQrOPYUp^nesp (279) in=40b,setxattr ]iPhefotctSa\\oRVq]olhX_almfgRQ\\QOkY[gmSi (981) in=40b,setxattr feSN]pkNedta[Xk]i[`VgdrXRsVQa[dW]RVOX\\ZO (1962) in=40b,setxattr ifgidpZ^lr`bXUTiiiYbkNr`q\\ijSr\\XRrafWcTd (349) in=40b,setxattr jYm`q[ekdUicbjN\\SZbc]rgUfjelbaZYQqR_S]\\m (1858) in=40b,setxattr Z^km\\Pb_aotanR[[sqak[kmTgkOehf`SjlXjNdnl (1717) in=40b,setxattr n[a[WQZgndTeT[PhPbsT_cp]cUajqmWetirrhSgt (911) in=40b,setxattr Y[[nPdmqa (806) in=9b,setxattr _eWfbeNenYaZSUkaYaa[Ud]oSVa\\e`_t[a]jYWjt (491) in=40b,setxattr \\OUqXocWimlYp]mSVRifacahfWfphmsUQt[tZf (806) in=38b] snapc 7=[7,5,4] ondisk+write+known_if_redirected e42)", 2020-03-09T00:25:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:51.848949+0000", 2020-03-09T00:25:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.031124224999999, 2020-03-09T00:25:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081625659999999996, 2020-03-09T00:25:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.936 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 228 2020-03-09T00:25:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848949+0000", 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.848949+0000", 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9226000000000002e-05 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.849048+0000", 2020-03-09T00:25:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8289999999999997e-06 2020-03-09T00:25:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.849057+0000", 2020-03-09T00:25:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2412000000000001e-05 2020-03-09T00:25:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.849069+0000", 2020-03-09T00:25:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9759999999999997e-06 2020-03-09T00:25:02.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.849073+0000", 2020-03-09T00:25:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6012e-05 2020-03-09T00:25:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.849109+0000", 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2490000000000004e-06 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.849117+0000", 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014612939999999999 2020-03-09T00:25:02.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.850578+0000", 2020-03-09T00:25:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109139 2020-03-09T00:25:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.850687+0000", 2020-03-09T00:25:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076763199999999997 2020-03-09T00:25:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:02.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.851455+0000", 2020-03-09T00:25:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00090080200000000003 2020-03-09T00:25:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.852355+0000", 2020-03-09T00:25:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046990759999999999 2020-03-09T00:25:02.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.857055+0000", 2020-03-09T00:25:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6919000000000002e-05 2020-03-09T00:25:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:51.857111+0000", 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081625659999999996 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.947 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:251 3.0s0 3:0efbb6fa:::smithi10428818-44:head [copy-from ver 5] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.0310878490000004, 2020-03-09T00:25:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001, 2020-03-09T00:25:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.948 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 251 2020-03-09T00:25:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6965999999999995e-05 2020-03-09T00:25:02.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849062+0000", 2020-03-09T00:25:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6689999999999995e-06 2020-03-09T00:25:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849071+0000", 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7941999999999999e-05 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849089+0000", 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9480000000000003e-06 2020-03-09T00:25:02.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849099+0000", 2020-03-09T00:25:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4438000000000003e-05 2020-03-09T00:25:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849153+0000", 2020-03-09T00:25:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012488000000000001 2020-03-09T00:25:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849278+0000", 2020-03-09T00:25:02.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013412429999999999 2020-03-09T00:25:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862691+0000", 2020-03-09T00:25:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011733578 2020-03-09T00:25:02.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.874424+0000", 2020-03-09T00:25:02.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000863506 2020-03-09T00:25:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875288+0000", 2020-03-09T00:25:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9488999999999998e-05 2020-03-09T00:25:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875357+0000", 2020-03-09T00:25:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001 2020-03-09T00:25:02.958 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:252 3.0s0 3:0efbb6fa:::smithi10428818-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.0309390510000007, 2020-03-09T00:25:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368, 2020-03-09T00:25:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 252 2020-03-09T00:25:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:02.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:02.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7280999999999998e-05 2020-03-09T00:25:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849171+0000", 2020-03-09T00:25:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6030000000000001e-06 2020-03-09T00:25:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849178+0000", 2020-03-09T00:25:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.308e-05 2020-03-09T00:25:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849191+0000", 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.553e-06 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849197+0000", 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030484499999999999 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849501+0000", 2020-03-09T00:25:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5923e-05 2020-03-09T00:25:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:02.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849557+0000", 2020-03-09T00:25:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019572935999999999 2020-03-09T00:25:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869130+0000", 2020-03-09T00:25:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013702000000000001 2020-03-09T00:25:02.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869267+0000", 2020-03-09T00:25:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0068816229999999999 2020-03-09T00:25:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876149+0000", 2020-03-09T00:25:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8204999999999999e-05 2020-03-09T00:25:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876247+0000", 2020-03-09T00:25:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109299 2020-03-09T00:25:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876356+0000", 2020-03-09T00:25:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368 2020-03-09T00:25:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:255 3.0s0 3:05d2883d:::smithi10428818-38:head [read 0~622592,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-03-09T00:25:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.862171+0000", 2020-03-09T00:25:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.0179020120000004, 2020-03-09T00:25:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014633525, 2020-03-09T00:25:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:02.973 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 255 2020-03-09T00:25:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862171+0000", 2020-03-09T00:25:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862171+0000", 2020-03-09T00:25:02.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3713000000000003e-05 2020-03-09T00:25:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862255+0000", 2020-03-09T00:25:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6209999999999999e-06 2020-03-09T00:25:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862264+0000", 2020-03-09T00:25:02.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5291e-05 2020-03-09T00:25:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862280+0000", 2020-03-09T00:25:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2049999999999996e-06 2020-03-09T00:25:02.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862287+0000", 2020-03-09T00:25:02.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0070196219999999997 2020-03-09T00:25:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869307+0000", 2020-03-09T00:25:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012668400000000001 2020-03-09T00:25:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869433+0000", 2020-03-09T00:25:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073713889999999999 2020-03-09T00:25:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876805+0000", 2020-03-09T00:25:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014633525 2020-03-09T00:25:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.983 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:256 3.0s0 3:05d2883d:::smithi10428818-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-03-09T00:25:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.862349+0000", 2020-03-09T00:25:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.0177239880000002, 2020-03-09T00:25:02.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014984402000000001, 2020-03-09T00:25:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 256 2020-03-09T00:25:02.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862349+0000", 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862349+0000", 2020-03-09T00:25:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0523000000000001e-05 2020-03-09T00:25:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862430+0000", 2020-03-09T00:25:02.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5809999999999996e-06 2020-03-09T00:25:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862435+0000", 2020-03-09T00:25:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.969e-06 2020-03-09T00:25:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862444+0000", 2020-03-09T00:25:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1849999999999998e-06 2020-03-09T00:25:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862451+0000", 2020-03-09T00:25:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075649549999999999 2020-03-09T00:25:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.870016+0000", 2020-03-09T00:25:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114026 2020-03-09T00:25:02.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.870130+0000", 2020-03-09T00:25:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072031630000000003 2020-03-09T00:25:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.877333+0000", 2020-03-09T00:25:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014984402000000001 2020-03-09T00:25:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:02.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:257 3.0s0 3:05d2883d:::smithi10428818-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-03-09T00:25:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.862483+0000", 2020-03-09T00:25:02.994 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.0175895849999996, 2020-03-09T00:25:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015213811000000001, 2020-03-09T00:25:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 257 2020-03-09T00:25:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.996 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862483+0000", 2020-03-09T00:25:02.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:02.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862483+0000", 2020-03-09T00:25:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3189000000000003e-05 2020-03-09T00:25:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:02.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862527+0000", 2020-03-09T00:25:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9249999999999997e-06 2020-03-09T00:25:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862531+0000", 2020-03-09T00:25:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5580000000000003e-06 2020-03-09T00:25:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862538+0000", 2020-03-09T00:25:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9519999999999999e-06 2020-03-09T00:25:03.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862541+0000", 2020-03-09T00:25:03.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0082563189999999998 2020-03-09T00:25:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.870797+0000", 2020-03-09T00:25:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109663 2020-03-09T00:25:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.870907+0000", 2020-03-09T00:25:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0067902049999999997 2020-03-09T00:25:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.877697+0000", 2020-03-09T00:25:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015213811000000001 2020-03-09T00:25:03.007 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.008 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:282 3.0s0 3:05d2883d:::smithi10428818-38:head [read 0~622592,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e49)", 2020-03-09T00:25:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.769313+0000", 2020-03-09T00:25:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.1107605769999997, 2020-03-09T00:25:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011902934, 2020-03-09T00:25:03.009 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 282 2020-03-09T00:25:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769313+0000", 2020-03-09T00:25:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769313+0000", 2020-03-09T00:25:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7104999999999997e-05 2020-03-09T00:25:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769360+0000", 2020-03-09T00:25:03.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.577e-06 2020-03-09T00:25:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769369+0000", 2020-03-09T00:25:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3910999999999999e-05 2020-03-09T00:25:03.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769383+0000", 2020-03-09T00:25:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.4409999999999999e-06 2020-03-09T00:25:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769393+0000", 2020-03-09T00:25:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1775999999999998e-05 2020-03-09T00:25:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769474+0000", 2020-03-09T00:25:03.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3318e-05 2020-03-09T00:25:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769488+0000", 2020-03-09T00:25:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002796116 2020-03-09T00:25:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.772284+0000", 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014832699999999999 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.772432+0000", 2020-03-09T00:25:03.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087833630000000006 2020-03-09T00:25:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.781215+0000", 2020-03-09T00:25:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011902934 2020-03-09T00:25:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.019 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:283 3.0s0 3:05d2883d:::smithi10428818-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected e49)", 2020-03-09T00:25:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.769435+0000", 2020-03-09T00:25:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.1106376789999999, 2020-03-09T00:25:03.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012799087000000001, 2020-03-09T00:25:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:03.021 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 283 2020-03-09T00:25:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769435+0000", 2020-03-09T00:25:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769435+0000", 2020-03-09T00:25:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2339999999999998e-05 2020-03-09T00:25:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769478+0000", 2020-03-09T00:25:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2220000000000004e-06 2020-03-09T00:25:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769482+0000", 2020-03-09T00:25:03.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2420000000000003e-06 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769489+0000", 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8139999999999998e-06 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769493+0000", 2020-03-09T00:25:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6770999999999999e-05 2020-03-09T00:25:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769590+0000", 2020-03-09T00:25:03.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9080999999999999e-05 2020-03-09T00:25:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769609+0000", 2020-03-09T00:25:03.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003503557 2020-03-09T00:25:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.773112+0000", 2020-03-09T00:25:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000112698 2020-03-09T00:25:03.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.773225+0000", 2020-03-09T00:25:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0090093619999999999 2020-03-09T00:25:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782234+0000", 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012799087000000001 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:284 3.0s0 3:05d2883d:::smithi10428818-38:head [read 0~622592,checksum] snapc 0=[] ondisk+read+known_if_redirected e49)", 2020-03-09T00:25:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.769527+0000", 2020-03-09T00:25:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.1105456199999999, 2020-03-09T00:25:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013328227999999999, 2020-03-09T00:25:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:03.032 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 284 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769527+0000", 2020-03-09T00:25:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769527+0000", 2020-03-09T00:25:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3214000000000003e-05 2020-03-09T00:25:03.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769571+0000", 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9439999999999998e-06 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769575+0000", 2020-03-09T00:25:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5429999999999996e-06 2020-03-09T00:25:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769582+0000", 2020-03-09T00:25:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3459999999999998e-06 2020-03-09T00:25:03.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769586+0000", 2020-03-09T00:25:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0908000000000006e-05 2020-03-09T00:25:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769656+0000", 2020-03-09T00:25:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6600000000000007e-06 2020-03-09T00:25:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769666+0000", 2020-03-09T00:25:03.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040895810000000001 2020-03-09T00:25:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.773756+0000", 2020-03-09T00:25:03.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012312400000000001 2020-03-09T00:25:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.773879+0000", 2020-03-09T00:25:03.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0089769080000000005 2020-03-09T00:25:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782856+0000", 2020-03-09T00:25:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013328227999999999 2020-03-09T00:25:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.041 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:285 3.9s0 3:9945b330:::smithi10428818-34:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.769623+0000", 2020-03-09T00:25:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.1104500799999997, 2020-03-09T00:25:03.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016923824000000001, 2020-03-09T00:25:03.043 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.043 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.043 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.043 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.043 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.043 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 285 2020-03-09T00:25:03.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.044 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769623+0000", 2020-03-09T00:25:03.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769623+0000", 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3260999999999998e-05 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769666+0000", 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0442999999999999e-05 2020-03-09T00:25:03.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769707+0000", 2020-03-09T00:25:03.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4482e-05 2020-03-09T00:25:03.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769721+0000", 2020-03-09T00:25:03.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4720000000000003e-06 2020-03-09T00:25:03.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769727+0000", 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9706000000000001e-05 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769786+0000", 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1578e-05 2020-03-09T00:25:03.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:03.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769798+0000", 2020-03-09T00:25:03.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003656117 2020-03-09T00:25:03.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.773454+0000", 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012591600000000001 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.773580+0000", 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012537169999999999 2020-03-09T00:25:03.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.774834+0000", 2020-03-09T00:25:03.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0065671810000000001 2020-03-09T00:25:03.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.781401+0000", 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.004987773 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.786389+0000", 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000158178 2020-03-09T00:25:03.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.786547+0000", 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016923824000000001 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:286 3.9s0 3:9945b330:::smithi10428818-34:head [rollback 4] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.769779+0000", 2020-03-09T00:25:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.1102942259999997, 2020-03-09T00:25:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018321638000000001, 2020-03-09T00:25:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 286 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769779+0000", 2020-03-09T00:25:03.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769779+0000", 2020-03-09T00:25:03.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0039000000000001e-05 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769819+0000", 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2300000000000008e-06 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769827+0000", 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4649999999999994e-06 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769836+0000", 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5369999999999999e-06 2020-03-09T00:25:03.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769840+0000", 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8940999999999997e-05 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769899+0000", 2020-03-09T00:25:03.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0766e-05 2020-03-09T00:25:03.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:03.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769910+0000", 2020-03-09T00:25:03.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059035759999999998 2020-03-09T00:25:03.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.775813+0000", 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00055491600000000002 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.776368+0000", 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010024350000000001 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.777371+0000", 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040332859999999996 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.781404+0000", 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0065780459999999997 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.787982+0000", 2020-03-09T00:25:03.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011840099999999999 2020-03-09T00:25:03.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788100+0000", 2020-03-09T00:25:03.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018321638000000001 2020-03-09T00:25:03.067 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.067 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:287 3.9s0 3:9945b330:::smithi10428818-34:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.769896+0000", 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.110176987, 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019042705, 2020-03-09T00:25:03.068 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 287 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769896+0000", 2020-03-09T00:25:03.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769896+0000", 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5759000000000001e-05 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769932+0000", 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3660000000000002e-06 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769936+0000", 2020-03-09T00:25:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9069999999999996e-06 2020-03-09T00:25:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769945+0000", 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7469999999999999e-06 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769949+0000", 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6261000000000003e-05 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769995+0000", 2020-03-09T00:25:03.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0147e-05 2020-03-09T00:25:03.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:03.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.770005+0000", 2020-03-09T00:25:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0079941419999999992 2020-03-09T00:25:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.777999+0000", 2020-03-09T00:25:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013511900000000001 2020-03-09T00:25:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.778135+0000", 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079071299999999996 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.778925+0000", 2020-03-09T00:25:03.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034493079999999999 2020-03-09T00:25:03.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782375+0000", 2020-03-09T00:25:03.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0064754519999999996 2020-03-09T00:25:03.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:03.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788850+0000", 2020-03-09T00:25:03.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8783999999999996e-05 2020-03-09T00:25:03.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.788939+0000", 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019042705 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.079 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:288 3.9s0 3:9d0ced77:::smithi10428818-33:head [delete] snapc f=[f,c,5,4] ondisk+write+peerstat_old+known_if_redirected e49)", 2020-03-09T00:25:03.080 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.769993+0000", 2020-03-09T00:25:03.080 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.1100800079999997, 2020-03-09T00:25:03.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020113079999999998, 2020-03-09T00:25:03.080 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.080 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.080 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 288 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769993+0000", 2020-03-09T00:25:03.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.769993+0000", 2020-03-09T00:25:03.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1789000000000003e-05 2020-03-09T00:25:03.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.770035+0000", 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0849999999999999e-06 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.770039+0000", 2020-03-09T00:25:03.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.9119999999999998e-06 2020-03-09T00:25:03.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.770047+0000", 2020-03-09T00:25:03.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5779999999999999e-06 2020-03-09T00:25:03.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.770050+0000", 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5414999999999999e-05 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.770086+0000", 2020-03-09T00:25:03.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3009999999999997e-06 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.770091+0000", 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0092526740000000007 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.779344+0000", 2020-03-09T00:25:03.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013616499999999999 2020-03-09T00:25:03.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.779480+0000", 2020-03-09T00:25:03.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00099429599999999998 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.780474+0000", 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019030780000000001 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.782377+0000", 2020-03-09T00:25:03.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0076449400000000002 2020-03-09T00:25:03.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:03.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790022+0000", 2020-03-09T00:25:03.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3846999999999999e-05 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.790106+0000", 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020113079999999998 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.090 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.091 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:297 3.9s0 3:9f137ce6:::smithi10428818-22:head [copy-from ver 10] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:03.091 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.818035+0000", 2020-03-09T00:25:03.091 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.0620384559999998, 2020-03-09T00:25:03.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022078066, 2020-03-09T00:25:03.091 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 297 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818035+0000", 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818035+0000", 2020-03-09T00:25:03.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6760000000000004e-05 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818111+0000", 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8480000000000007e-06 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818120+0000", 2020-03-09T00:25:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5758999999999999e-05 2020-03-09T00:25:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818136+0000", 2020-03-09T00:25:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7700000000000001e-06 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818144+0000", 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1414000000000005e-05 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818205+0000", 2020-03-09T00:25:03.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013875500000000001 2020-03-09T00:25:03.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818344+0000", 2020-03-09T00:25:03.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.009793922 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.828138+0000", 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046157439999999997 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.832754+0000", 2020-03-09T00:25:03.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072716919999999997 2020-03-09T00:25:03.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:03.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.840025+0000", 2020-03-09T00:25:03.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7402000000000001e-05 2020-03-09T00:25:03.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.840113+0000", 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022078066 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:298 3.9s0 3:9f137ce6:::smithi10428818-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:03.101 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.818198+0000", 2020-03-09T00:25:03.102 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.0618745890000003, 2020-03-09T00:25:03.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022669233, 2020-03-09T00:25:03.102 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.102 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:03.102 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:03.102 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:03.102 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 298 2020-03-09T00:25:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818198+0000", 2020-03-09T00:25:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818198+0000", 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5738000000000002e-05 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818244+0000", 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9400000000000008e-06 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818253+0000", 2020-03-09T00:25:03.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2584e-05 2020-03-09T00:25:03.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818266+0000", 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6489999999999996e-06 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818271+0000", 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00025771399999999998 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818529+0000", 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2994999999999998e-05 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.818562+0000", 2020-03-09T00:25:03.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014069296 2020-03-09T00:25:03.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.109 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.109 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.832631+0000", 2020-03-09T00:25:03.109 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1799999999999997e-05 2020-03-09T00:25:03.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.109 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.832703+0000", 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0079557440000000007 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.840659+0000", 2020-03-09T00:25:03.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000110074 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.840769+0000", 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8698999999999998e-05 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.840868+0000", 2020-03-09T00:25:03.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022669233 2020-03-09T00:25:03.112 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.112 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.112 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.113 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.cs3 49/32 ECSubWrite(tid=77, reqid=client.4751.0:301, at_version=49'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:03.113 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.831108+0000", 2020-03-09T00:25:03.113 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.0489653309999998, 2020-03-09T00:25:03.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0095726170000000003, 2020-03-09T00:25:03.113 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831108+0000", 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:03.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831108+0000", 2020-03-09T00:25:03.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00040621199999999998 2020-03-09T00:25:03.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:03.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831514+0000", 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.372e-05 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831538+0000", 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5104e-05 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:03.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831553+0000", 2020-03-09T00:25:03.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1040000000000003e-06 2020-03-09T00:25:03.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831559+0000", 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0485000000000001e-05 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831619+0000", 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4336e-05 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.831634+0000", 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0089887679999999994 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.840622+0000", 2020-03-09T00:25:03.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7887999999999999e-05 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.840680+0000", 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0095726170000000003 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.120 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.121 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:03.121 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:03.121 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:03.123 INFO:tasks.rados.rados.0.smithi104.stdout:146: writing smithi10428818-18 from 1294336 to 1802240 tid 1 2020-03-09T00:25:03.123 INFO:tasks.rados.rados.0.smithi104.stdout:146: writing smithi10428818-18 from 1802240 to 2310144 tid 2 2020-03-09T00:25:03.123 INFO:tasks.rados.rados.0.smithi104.stdout:146: writing smithi10428818-18 from 2310144 to 2572288 tid 3 2020-03-09T00:25:03.123 INFO:tasks.rados.rados.0.smithi104.stdout:147: snap_create 2020-03-09T00:25:03.123 INFO:tasks.rados.rados.0.smithi104.stdout:145: expect (ObjNum 31 snap 0 seq_num 31) 2020-03-09T00:25:03.124 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 44 v 19 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-03-09T00:25:03.124 INFO:tasks.rados.rados.0.smithi104.stdout:146: finishing write tid 1 to smithi10428818-18 2020-03-09T00:25:03.124 INFO:tasks.rados.rados.0.smithi104.stdout:146: finishing write tid 2 to smithi10428818-18 2020-03-09T00:25:03.124 INFO:tasks.rados.rados.0.smithi104.stdout:146: finishing write tid 3 to smithi10428818-18 2020-03-09T00:25:03.124 INFO:tasks.rados.rados.0.smithi104.stdout:146: finishing write tid 5 to smithi10428818-18 2020-03-09T00:25:03.124 INFO:tasks.rados.rados.0.smithi104.stdout:132: finishing rollback tid 1 to smithi10428818-49 2020-03-09T00:25:03.124 INFO:tasks.rados.rados.0.smithi104.stdout:146: finishing write tid 6 to smithi10428818-18 2020-03-09T00:25:03.125 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 18 v 31 (ObjNum 76 snap 11 seq_num 76) dirty exists 2020-03-09T00:25:03.125 INFO:tasks.rados.rados.0.smithi104.stdout:146: left oid 18 (ObjNum 76 snap 11 seq_num 76) 2020-03-09T00:25:03.125 INFO:tasks.rados.rados.0.smithi104.stdout:132: finishing rollback tid 2 to smithi10428818-49 2020-03-09T00:25:03.125 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 49 v 21 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-03-09T00:25:03.125 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:25:03.126 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:03.121+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:03.223 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:03.222+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:03.241 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:03.241 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:03.242 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:03.242 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:03.250 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:25:03.324 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:03.323+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:03.365 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:03.365 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:03.365 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:03.365 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:03.365 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:03.373 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:25:03.425 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:03.424+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:03.527 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:03.525+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:03.528 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:03.529 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:03.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:03.529 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:03.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.529 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:196 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~540672] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:03.529 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:03.530 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.700497954999999, 2020-03-09T00:25:03.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501, 2020-03-09T00:25:03.530 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.530 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 196 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035415000000000002 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826046+0000", 2020-03-09T00:25:03.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2260000000000005e-06 2020-03-09T00:25:03.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826054+0000", 2020-03-09T00:25:03.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3873000000000001e-05 2020-03-09T00:25:03.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826068+0000", 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826074+0000", 2020-03-09T00:25:03.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0126e-05 2020-03-09T00:25:03.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826134+0000", 2020-03-09T00:25:03.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001117 2020-03-09T00:25:03.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826246+0000", 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00124165 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827488+0000", 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088469059999999999 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836335+0000", 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035733999999999999 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:03.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836692+0000", 2020-03-09T00:25:03.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1440000000000005e-05 2020-03-09T00:25:03.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836783+0000", 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:197 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:03.539 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:03.540 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.695239418, 2020-03-09T00:25:03.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097135529999999998, 2020-03-09T00:25:03.540 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.540 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.540 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.540 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.540 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.541 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 197 2020-03-09T00:25:03.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.541 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:03.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027437700000000002 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831225+0000", 2020-03-09T00:25:03.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8239999999999996e-06 2020-03-09T00:25:03.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831231+0000", 2020-03-09T00:25:03.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3345e-05 2020-03-09T00:25:03.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831244+0000", 2020-03-09T00:25:03.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4499999999999998e-06 2020-03-09T00:25:03.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831251+0000", 2020-03-09T00:25:03.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057306900000000005 2020-03-09T00:25:03.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831824+0000", 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103781 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831928+0000", 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000811518 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832740+0000", 2020-03-09T00:25:03.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036368210000000002 2020-03-09T00:25:03.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836377+0000", 2020-03-09T00:25:03.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042093749999999996 2020-03-09T00:25:03.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:03.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840586+0000", 2020-03-09T00:25:03.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7992999999999997e-05 2020-03-09T00:25:03.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840664+0000", 2020-03-09T00:25:03.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097135529999999998 2020-03-09T00:25:03.549 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.549 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.549 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.550 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:198 3.5s0 3:ada90060:::smithi10428818-8:head [setxattr _header (55) in=7b] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:03.550 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:03.550 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.694883733999999, 2020-03-09T00:25:03.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095424670000000007, 2020-03-09T00:25:03.550 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:03.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9859999999999999e-05 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831376+0000", 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3319999999999997e-06 2020-03-09T00:25:03.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831382+0000", 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1367e-05 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831394+0000", 2020-03-09T00:25:03.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4580000000000003e-06 2020-03-09T00:25:03.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831398+0000", 2020-03-09T00:25:03.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039429479999999999 2020-03-09T00:25:03.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835341+0000", 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011724 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835458+0000", 2020-03-09T00:25:03.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068325099999999995 2020-03-09T00:25:03.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836141+0000", 2020-03-09T00:25:03.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092122099999999995 2020-03-09T00:25:03.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.837063+0000", 2020-03-09T00:25:03.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036999060000000002 2020-03-09T00:25:03.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840763+0000", 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5884000000000006e-05 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840848+0000", 2020-03-09T00:25:03.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095424670000000007 2020-03-09T00:25:03.560 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.560 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.560 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.560 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:199 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:03.560 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.694751836, 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149, 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.561 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 199 2020-03-09T00:25:03.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.562 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:03.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:03.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1242999999999998e-05 2020-03-09T00:25:03.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831479+0000", 2020-03-09T00:25:03.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5740000000000001e-06 2020-03-09T00:25:03.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831486+0000", 2020-03-09T00:25:03.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1513e-05 2020-03-09T00:25:03.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831497+0000", 2020-03-09T00:25:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2760000000000002e-06 2020-03-09T00:25:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831502+0000", 2020-03-09T00:25:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053448499999999999 2020-03-09T00:25:03.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836846+0000", 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103844 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:03.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836950+0000", 2020-03-09T00:25:03.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045940299999999998 2020-03-09T00:25:03.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841544+0000", 2020-03-09T00:25:03.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105018 2020-03-09T00:25:03.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841649+0000", 2020-03-09T00:25:03.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002376801 2020-03-09T00:25:03.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.844026+0000", 2020-03-09T00:25:03.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149 2020-03-09T00:25:03.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.580 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.581 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.8s5 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:25:03.582 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.837935+0000", 2020-03-09T00:25:03.582 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.688254769, 2020-03-09T00:25:03.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050432209999999996, 2020-03-09T00:25:03.582 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.582 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.582 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.837935+0000", 2020-03-09T00:25:03.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.837935+0000", 2020-03-09T00:25:03.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000267466 2020-03-09T00:25:03.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838202+0000", 2020-03-09T00:25:03.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5054e-05 2020-03-09T00:25:03.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838217+0000", 2020-03-09T00:25:03.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0169e-05 2020-03-09T00:25:03.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838228+0000", 2020-03-09T00:25:03.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0690000000000003e-06 2020-03-09T00:25:03.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838232+0000", 2020-03-09T00:25:03.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0906999999999999e-05 2020-03-09T00:25:03.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838283+0000", 2020-03-09T00:25:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6530000000000006e-06 2020-03-09T00:25:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.838292+0000", 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046448599999999998 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.842937+0000", 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1043e-05 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.842978+0000", 2020-03-09T00:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050432209999999996 2020-03-09T00:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.593 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as5 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:25:03.593 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.864378+0000", 2020-03-09T00:25:03.593 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.661811429, 2020-03-09T00:25:03.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054133530000000001, 2020-03-09T00:25:03.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864378+0000", 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864378+0000", 2020-03-09T00:25:03.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030868399999999999 2020-03-09T00:25:03.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864687+0000", 2020-03-09T00:25:03.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9913e-05 2020-03-09T00:25:03.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864717+0000", 2020-03-09T00:25:03.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7563000000000001e-05 2020-03-09T00:25:03.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864734+0000", 2020-03-09T00:25:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6340000000000001e-06 2020-03-09T00:25:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864742+0000", 2020-03-09T00:25:03.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6957000000000003e-05 2020-03-09T00:25:03.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864809+0000", 2020-03-09T00:25:03.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2982e-05 2020-03-09T00:25:03.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864822+0000", 2020-03-09T00:25:03.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049205280000000004 2020-03-09T00:25:03.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869743+0000", 2020-03-09T00:25:03.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9091999999999997e-05 2020-03-09T00:25:03.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.869792+0000", 2020-03-09T00:25:03.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054133530000000001 2020-03-09T00:25:03.601 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.601 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.601 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.0s2 42/32 ECSubRead(tid=51, to_read={3:054f30c0:::smithi10428818-49:head=0,204800,0}, subchunks={3:054f30c0:::smithi10428818-49:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:25:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.851318+0000", 2020-03-09T00:25:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.674871295999999, 2020-03-09T00:25:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082708869999999993, 2020-03-09T00:25:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851318+0000", 2020-03-09T00:25:03.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851318+0000", 2020-03-09T00:25:03.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0850999999999997e-05 2020-03-09T00:25:03.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851379+0000", 2020-03-09T00:25:03.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2201999999999999e-05 2020-03-09T00:25:03.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851391+0000", 2020-03-09T00:25:03.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7319000000000001e-05 2020-03-09T00:25:03.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851409+0000", 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1439999999999997e-06 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851416+0000", 2020-03-09T00:25:03.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000481676 2020-03-09T00:25:03.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851898+0000", 2020-03-09T00:25:03.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0076916950000000001 2020-03-09T00:25:03.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859589+0000", 2020-03-09T00:25:03.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082708869999999993 2020-03-09T00:25:03.608 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.608 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.608 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.609 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 42/32 ECSubWrite(tid=52, reqid=client.4751.0:228, at_version=42'17, trim_to=0'0, roll_forward_to=38'16))", 2020-03-09T00:25:03.609 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.851700+0000", 2020-03-09T00:25:03.609 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.674490198000001, 2020-03-09T00:25:03.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050820639999999999, 2020-03-09T00:25:03.609 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.610 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.610 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851700+0000", 2020-03-09T00:25:03.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851700+0000", 2020-03-09T00:25:03.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137498 2020-03-09T00:25:03.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851837+0000", 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.287e-05 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851860+0000", 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5835e-05 2020-03-09T00:25:03.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851876+0000", 2020-03-09T00:25:03.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6510000000000004e-06 2020-03-09T00:25:03.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851882+0000", 2020-03-09T00:25:03.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2135000000000002e-05 2020-03-09T00:25:03.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851925+0000", 2020-03-09T00:25:03.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0508e-05 2020-03-09T00:25:03.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851935+0000", 2020-03-09T00:25:03.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048068549999999996 2020-03-09T00:25:03.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.856742+0000", 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9712000000000002e-05 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.856782+0000", 2020-03-09T00:25:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050820639999999999 2020-03-09T00:25:03.617 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.617 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.617 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.617 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.0s2 42/32 ECSubRead(tid=53, to_read={3:054f30c0:::smithi10428818-49:head=0,204800,0}, subchunks={3:054f30c0:::smithi10428818-49:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:25:03.617 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.851965+0000", 2020-03-09T00:25:03.618 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.674225031000001, 2020-03-09T00:25:03.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0079110699999999992, 2020-03-09T00:25:03.618 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.618 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.618 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851965+0000", 2020-03-09T00:25:03.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.851965+0000", 2020-03-09T00:25:03.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2415e-05 2020-03-09T00:25:03.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852007+0000", 2020-03-09T00:25:03.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7320000000000001e-06 2020-03-09T00:25:03.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852014+0000", 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7820000000000005e-06 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852024+0000", 2020-03-09T00:25:03.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.305e-06 2020-03-09T00:25:03.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852029+0000", 2020-03-09T00:25:03.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0076273399999999998 2020-03-09T00:25:03.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859656+0000", 2020-03-09T00:25:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021949599999999999 2020-03-09T00:25:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859876+0000", 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0079110699999999992 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.0s2 42/32 ECSubRead(tid=54, to_read={3:054f30c0:::smithi10428818-49:head=0,204800,0}, subchunks={3:054f30c0:::smithi10428818-49:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:25:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:51.852777+0000", 2020-03-09T00:25:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.67341317, 2020-03-09T00:25:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007436804, 2020-03-09T00:25:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.625 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852777+0000", 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852777+0000", 2020-03-09T00:25:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5549e-05 2020-03-09T00:25:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852822+0000", 2020-03-09T00:25:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8220000000000002e-06 2020-03-09T00:25:03.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852832+0000", 2020-03-09T00:25:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2646000000000001e-05 2020-03-09T00:25:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852845+0000", 2020-03-09T00:25:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1239999999999998e-06 2020-03-09T00:25:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.852849+0000", 2020-03-09T00:25:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0070577160000000003 2020-03-09T00:25:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.859906+0000", 2020-03-09T00:25:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030694700000000001 2020-03-09T00:25:03.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:51.860213+0000", 2020-03-09T00:25:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007436804 2020-03-09T00:25:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.631 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.863480+0000", 2020-03-09T00:25:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6627102090000001, 2020-03-09T00:25:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053954390000000001, 2020-03-09T00:25:03.632 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.863480+0000", 2020-03-09T00:25:03.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.863480+0000", 2020-03-09T00:25:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052070899999999995 2020-03-09T00:25:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.864000+0000", 2020-03-09T00:25:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4573e-05 2020-03-09T00:25:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.864025+0000", 2020-03-09T00:25:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5559000000000001e-05 2020-03-09T00:25:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.864040+0000", 2020-03-09T00:25:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7329999999999997e-06 2020-03-09T00:25:03.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.864047+0000", 2020-03-09T00:25:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1927999999999994e-05 2020-03-09T00:25:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.864109+0000", 2020-03-09T00:25:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2858e-05 2020-03-09T00:25:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.864122+0000", 2020-03-09T00:25:03.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00468893 2020-03-09T00:25:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.868811+0000", 2020-03-09T00:25:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4148999999999999e-05 2020-03-09T00:25:03.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.868875+0000", 2020-03-09T00:25:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053954390000000001 2020-03-09T00:25:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.640 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/32 ECSubWrite(tid=22, reqid=client.4751.0:275, at_version=48'12, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:25:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:57.788672+0000", 2020-03-09T00:25:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7375177629999996, 2020-03-09T00:25:03.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00662423, 2020-03-09T00:25:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.788672+0000", 2020-03-09T00:25:03.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.788672+0000", 2020-03-09T00:25:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00041144700000000001 2020-03-09T00:25:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.789083+0000", 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7095000000000001e-05 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.789101+0000", 2020-03-09T00:25:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0915e-05 2020-03-09T00:25:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.789111+0000", 2020-03-09T00:25:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1250000000000004e-06 2020-03-09T00:25:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.789119+0000", 2020-03-09T00:25:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9582000000000001e-05 2020-03-09T00:25:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.789168+0000", 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7440000000000002e-06 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.789178+0000", 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060851179999999996 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.795263+0000", 2020-03-09T00:25:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3204000000000003e-05 2020-03-09T00:25:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.795296+0000", 2020-03-09T00:25:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00662423 2020-03-09T00:25:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.649 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 49/32 ECSubWrite(tid=74, reqid=client.4751.0:287, at_version=49'25, trim_to=0'0, roll_forward_to=48'21))", 2020-03-09T00:25:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.780125+0000", 2020-03-09T00:25:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7460646950000003, 2020-03-09T00:25:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051024979999999996, 2020-03-09T00:25:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.650 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780125+0000", 2020-03-09T00:25:03.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780125+0000", 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0763000000000001e-05 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780186+0000", 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0828000000000001e-05 2020-03-09T00:25:03.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780207+0000", 2020-03-09T00:25:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6076e-05 2020-03-09T00:25:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780223+0000", 2020-03-09T00:25:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8029999999999997e-06 2020-03-09T00:25:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780231+0000", 2020-03-09T00:25:03.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021020500000000001 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780441+0000", 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3834e-05 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.780455+0000", 2020-03-09T00:25:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004720798 2020-03-09T00:25:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.785175+0000", 2020-03-09T00:25:03.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2191000000000002e-05 2020-03-09T00:25:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.785228+0000", 2020-03-09T00:25:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051024979999999996 2020-03-09T00:25:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.657 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s3 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.792817+0000", 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.73337227000000005, 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052987240000000003, 2020-03-09T00:25:03.658 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792817+0000", 2020-03-09T00:25:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792817+0000", 2020-03-09T00:25:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068506800000000005 2020-03-09T00:25:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793503+0000", 2020-03-09T00:25:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9481000000000001e-05 2020-03-09T00:25:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793532+0000", 2020-03-09T00:25:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7968000000000001e-05 2020-03-09T00:25:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793550+0000", 2020-03-09T00:25:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.576e-06 2020-03-09T00:25:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793558+0000", 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8860000000000002e-05 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793616+0000", 2020-03-09T00:25:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2644e-05 2020-03-09T00:25:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793629+0000", 2020-03-09T00:25:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044318120000000003 2020-03-09T00:25:03.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798061+0000", 2020-03-09T00:25:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5315000000000001e-05 2020-03-09T00:25:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798116+0000", 2020-03-09T00:25:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052987240000000003 2020-03-09T00:25:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:346 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.70065823900000002, 2020-03-09T00:25:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654, 2020-03-09T00:25:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.667 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 346 2020-03-09T00:25:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:03.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014902900000000001 2020-03-09T00:25:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825681+0000", 2020-03-09T00:25:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:03.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825687+0000", 2020-03-09T00:25:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6099999999999995e-06 2020-03-09T00:25:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825696+0000", 2020-03-09T00:25:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6149999999999999e-06 2020-03-09T00:25:03.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825701+0000", 2020-03-09T00:25:03.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4975000000000002e-05 2020-03-09T00:25:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825746+0000", 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0597e-05 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825836+0000", 2020-03-09T00:25:03.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091597100000000002 2020-03-09T00:25:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826752+0000", 2020-03-09T00:25:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046291539999999999 2020-03-09T00:25:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831382+0000", 2020-03-09T00:25:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098416760000000006 2020-03-09T00:25:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:03.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841223+0000", 2020-03-09T00:25:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100926 2020-03-09T00:25:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841324+0000", 2020-03-09T00:25:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654 2020-03-09T00:25:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:347 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~622592] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:03.679 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.69339041700000004, 2020-03-09T00:25:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999, 2020-03-09T00:25:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.680 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 347 2020-03-09T00:25:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:03.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000469071 2020-03-09T00:25:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833268+0000", 2020-03-09T00:25:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7889999999999995e-06 2020-03-09T00:25:03.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833276+0000", 2020-03-09T00:25:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6033999999999999e-05 2020-03-09T00:25:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833292+0000", 2020-03-09T00:25:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8850000000000001e-06 2020-03-09T00:25:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833299+0000", 2020-03-09T00:25:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833343+0000", 2020-03-09T00:25:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105574 2020-03-09T00:25:03.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833448+0000", 2020-03-09T00:25:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094406100000000005 2020-03-09T00:25:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834392+0000", 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088904150000000005 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843283+0000", 2020-03-09T00:25:03.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052783500000000004 2020-03-09T00:25:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843811+0000", 2020-03-09T00:25:03.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1547999999999998e-05 2020-03-09T00:25:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843902+0000", 2020-03-09T00:25:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999 2020-03-09T00:25:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.690 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:348 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~147456] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:03.691 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.69283432700000003, 2020-03-09T00:25:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999, 2020-03-09T00:25:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.692 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 348 2020-03-09T00:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019096900000000001 2020-03-09T00:25:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833546+0000", 2020-03-09T00:25:03.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4060000000000002e-06 2020-03-09T00:25:03.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833553+0000", 2020-03-09T00:25:03.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1321000000000001e-05 2020-03-09T00:25:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6310000000000002e-06 2020-03-09T00:25:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833569+0000", 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037440260000000001 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837313+0000", 2020-03-09T00:25:03.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134988 2020-03-09T00:25:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837448+0000", 2020-03-09T00:25:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001155454 2020-03-09T00:25:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.838603+0000", 2020-03-09T00:25:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046829050000000002 2020-03-09T00:25:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843286+0000", 2020-03-09T00:25:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011927330000000001 2020-03-09T00:25:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844479+0000", 2020-03-09T00:25:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5680000000000005e-05 2020-03-09T00:25:03.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844575+0000", 2020-03-09T00:25:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999 2020-03-09T00:25:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.701 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:349 3.5s0 3:ada90060:::smithi10428818-8:head [setxattr _header (56) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.69256876700000003, 2020-03-09T00:25:03.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001, 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 349 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.703 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8177999999999997e-05 2020-03-09T00:25:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833689+0000", 2020-03-09T00:25:03.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3619999999999997e-06 2020-03-09T00:25:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833695+0000", 2020-03-09T00:25:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1154e-05 2020-03-09T00:25:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833706+0000", 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5030000000000002e-06 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833712+0000", 2020-03-09T00:25:03.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0079814919999999998 2020-03-09T00:25:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841694+0000", 2020-03-09T00:25:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013282600000000001 2020-03-09T00:25:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841826+0000", 2020-03-09T00:25:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071321600000000002 2020-03-09T00:25:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:03.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.842540+0000", 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074733700000000004 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843287+0000", 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001845368 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845132+0000", 2020-03-09T00:25:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000004e-05 2020-03-09T00:25:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845219+0000", 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:350 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.69243911000000002, 2020-03-09T00:25:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417, 2020-03-09T00:25:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:03.713 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 350 2020-03-09T00:25:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:03.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0074999999999998e-05 2020-03-09T00:25:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833791+0000", 2020-03-09T00:25:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1530000000000003e-06 2020-03-09T00:25:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833800+0000", 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3621000000000001e-05 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833813+0000", 2020-03-09T00:25:03.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9899999999999999e-06 2020-03-09T00:25:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833817+0000", 2020-03-09T00:25:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0093306459999999997 2020-03-09T00:25:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843148+0000", 2020-03-09T00:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012790100000000001 2020-03-09T00:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:03.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843276+0000", 2020-03-09T00:25:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025486789999999999 2020-03-09T00:25:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845825+0000", 2020-03-09T00:25:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120695 2020-03-09T00:25:03.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845945+0000", 2020-03-09T00:25:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033866569999999999 2020-03-09T00:25:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.849332+0000", 2020-03-09T00:25:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417 2020-03-09T00:25:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 53/48 ECSubWriteReply(tid=30, last_complete=53'18, committed=1, applied=1))", 2020-03-09T00:25:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.834035+0000", 2020-03-09T00:25:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.692155242, 2020-03-09T00:25:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007049563, 2020-03-09T00:25:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:03.723 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834035+0000", 2020-03-09T00:25:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834035+0000", 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1619999999999997e-05 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834086+0000", 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1999999999999997e-06 2020-03-09T00:25:03.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834093+0000", 2020-03-09T00:25:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3548e-05 2020-03-09T00:25:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834107+0000", 2020-03-09T00:25:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.668e-06 2020-03-09T00:25:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834112+0000", 2020-03-09T00:25:03.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069546820000000002 2020-03-09T00:25:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841066+0000", 2020-03-09T00:25:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7844999999999999e-05 2020-03-09T00:25:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:03.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841084+0000", 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007049563 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:03.729 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:03.730 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:03.731 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:03.729+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:03.732 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:25:03.791 INFO:tasks.rados.rados.0.smithi104.stdout:132: done (4 left) 2020-03-09T00:25:03.791 INFO:tasks.rados.rados.0.smithi104.stdout:144: done (3 left) 2020-03-09T00:25:03.791 INFO:tasks.rados.rados.0.smithi104.stdout:145: done (2 left) 2020-03-09T00:25:03.791 INFO:tasks.rados.rados.0.smithi104.stdout:146: done (1 left) 2020-03-09T00:25:03.792 INFO:tasks.rados.rados.0.smithi104.stdout:147: done (0 left) 2020-03-09T00:25:03.792 INFO:tasks.rados.rados.0.smithi104.stdout:148: setattr oid 2 current snap is 12 2020-03-09T00:25:03.794 INFO:tasks.rados.rados.0.smithi104.stdout:149: read oid 16 snap 11 2020-03-09T00:25:03.794 INFO:tasks.rados.rados.0.smithi104.stdout:149: expect (ObjNum 70 snap 11 seq_num 70) 2020-03-09T00:25:03.794 INFO:tasks.rados.rados.0.smithi104.stdout:150: read oid 20 snap -1 2020-03-09T00:25:03.795 INFO:tasks.rados.rados.0.smithi104.stdout:150: expect (ObjNum 73 snap 11 seq_num 73) 2020-03-09T00:25:03.795 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 17 current snap is 12 2020-03-09T00:25:03.795 INFO:tasks.rados.rados.0.smithi104.stdout:151: seq_num 77 ranges {524288=524288,1048576=524288,1572864=245760} 2020-03-09T00:25:03.801 INFO:tasks.rados.rados.0.smithi104.stdout:151: writing smithi10428818-17 from 524288 to 1048576 tid 1 2020-03-09T00:25:03.804 INFO:tasks.rados.rados.0.smithi104.stdout:151: writing smithi10428818-17 from 1048576 to 1572864 tid 2 2020-03-09T00:25:03.806 INFO:tasks.rados.rados.0.smithi104.stdout:151: writing smithi10428818-17 from 1572864 to 1818624 tid 3 2020-03-09T00:25:03.806 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 27 current snap is 12 2020-03-09T00:25:03.806 INFO:tasks.rados.rados.0.smithi104.stdout:152: seq_num 78 ranges {442368=557056,999424=671744,1671168=98304} 2020-03-09T00:25:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is set 2020-03-09T00:25:03.812 INFO:tasks.rados.rados.0.smithi104.stdout:152: writing smithi10428818-27 from 442368 to 999424 tid 1 2020-03-09T00:25:03.817 INFO:tasks.rados.rados.0.smithi104.stdout:152: writing smithi10428818-27 from 999424 to 1671168 tid 2 2020-03-09T00:25:03.818 INFO:tasks.rados.rados.0.smithi104.stdout:152: writing smithi10428818-27 from 1671168 to 1769472 tid 3 2020-03-09T00:25:03.818 INFO:tasks.rados.rados.0.smithi104.stdout:153: read oid 10 snap -1 2020-03-09T00:25:03.819 INFO:tasks.rados.rados.0.smithi104.stdout:153: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:25:03.819 INFO:tasks.rados.rados.0.smithi104.stdout:154: read oid 18 snap 10 2020-03-09T00:25:03.819 INFO:tasks.rados.rados.0.smithi104.stdout:154: expect (ObjNum 59 snap 6 seq_num 59) 2020-03-09T00:25:03.819 INFO:tasks.rados.rados.0.smithi104.stdout:155: read oid 39 snap 9 2020-03-09T00:25:03.819 INFO:tasks.rados.rados.0.smithi104.stdout:155: expect (ObjNum 38 snap 0 seq_num 38) 2020-03-09T00:25:03.819 INFO:tasks.rados.rados.0.smithi104.stdout:156: read oid 24 snap -1 2020-03-09T00:25:03.819 INFO:tasks.rados.rados.0.smithi104.stdout:156: expect (ObjNum 23 snap 0 seq_num 23) 2020-03-09T00:25:03.820 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 23 current snap is 12 2020-03-09T00:25:03.820 INFO:tasks.rados.rados.0.smithi104.stdout:157: seq_num 79 ranges {1146880=573440,1720320=425984,2146304=344064} 2020-03-09T00:25:03.828 INFO:tasks.rados.rados.0.smithi104.stdout:157: writing smithi10428818-23 from 1146880 to 1720320 tid 1 2020-03-09T00:25:03.831 INFO:tasks.rados.rados.0.smithi104.stdout:157: writing smithi10428818-23 from 1720320 to 2146304 tid 2 2020-03-09T00:25:03.831 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:03.830+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:03.833 INFO:tasks.rados.rados.0.smithi104.stdout:157: writing smithi10428818-23 from 2146304 to 2490368 tid 3 2020-03-09T00:25:03.833 INFO:tasks.rados.rados.0.smithi104.stdout:158: rmattr oid 42 current snap is 12 2020-03-09T00:25:03.834 INFO:tasks.rados.rados.0.smithi104.stdout:158: done (10 left) 2020-03-09T00:25:03.834 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 29 current snap is 12 2020-03-09T00:25:03.834 INFO:tasks.rados.rados.0.smithi104.stdout:159: seq_num 80 ranges {475136=589824,1064960=573440,1638400=196608} 2020-03-09T00:25:03.839 INFO:tasks.rados.rados.0.smithi104.stdout:159: writing smithi10428818-29 from 475136 to 1064960 tid 1 2020-03-09T00:25:03.842 INFO:tasks.rados.rados.0.smithi104.stdout:159: writing smithi10428818-29 from 1064960 to 1638400 tid 2 2020-03-09T00:25:03.844 INFO:tasks.rados.rados.0.smithi104.stdout:159: writing smithi10428818-29 from 1638400 to 1835008 tid 3 2020-03-09T00:25:03.844 INFO:tasks.rados.rados.0.smithi104.stdout:160: copy_from oid 19 from oid 15 current snap is 12 2020-03-09T00:25:03.844 INFO:tasks.rados.rados.0.smithi104.stdout:161: read oid 42 snap -1 2020-03-09T00:25:03.844 INFO:tasks.rados.rados.0.smithi104.stdout:161: expect (ObjNum 71 snap 11 seq_num 71) 2020-03-09T00:25:03.845 INFO:tasks.rados.rados.0.smithi104.stdout:162: copy_from oid 49 from oid 9 current snap is 12 2020-03-09T00:25:03.845 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 2 v 44 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-03-09T00:25:03.845 INFO:tasks.rados.rados.0.smithi104.stdout:149: expect (ObjNum 70 snap 11 seq_num 70) 2020-03-09T00:25:03.870 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:397 3.7s0 3.f05ab707 (undecoded) ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:03.870 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.040598216999999999, 2020-03-09T00:25:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.040618802000000002, 2020-03-09T00:25:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "delayed", 2020-03-09T00:25:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.871 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 397 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015862319999999999 2020-03-09T00:25:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:03.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829895+0000", 2020-03-09T00:25:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1249999999999999e-06 2020-03-09T00:25:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:03.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829904+0000", 2020-03-09T00:25:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5704000000000001e-05 2020-03-09T00:25:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829920+0000", 2020-03-09T00:25:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3140000000000004e-06 2020-03-09T00:25:03.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829928+0000", 2020-03-09T00:25:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000122717 2020-03-09T00:25:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830051+0000", 2020-03-09T00:25:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9819999999999999e-06 2020-03-09T00:25:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:03.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830056+0000", 2020-03-09T00:25:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017470739999999999 2020-03-09T00:25:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.878 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:398 3.7s0 3.f05ab707 (undecoded) ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.038102773, 2020-03-09T00:25:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.038170822, 2020-03-09T00:25:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "delayed", 2020-03-09T00:25:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:03.879 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 398 2020-03-09T00:25:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:03.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051814600000000004 2020-03-09T00:25:03.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831322+0000", 2020-03-09T00:25:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0118e-05 2020-03-09T00:25:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831333+0000", 2020-03-09T00:25:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0710999999999999e-05 2020-03-09T00:25:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:03.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831353+0000", 2020-03-09T00:25:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7870000000000004e-06 2020-03-09T00:25:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831360+0000", 2020-03-09T00:25:03.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013810100000000001 2020-03-09T00:25:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831498+0000", 2020-03-09T00:25:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3380000000000001e-06 2020-03-09T00:25:03.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831502+0000", 2020-03-09T00:25:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00069720100000000005 2020-03-09T00:25:03.886 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:399 3.7s0 3.f05ab707 (undecoded) ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:03.887 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.036001599000000002, 2020-03-09T00:25:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.036109859000000001, 2020-03-09T00:25:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "delayed", 2020-03-09T00:25:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.888 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 399 2020-03-09T00:25:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:03.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00031358200000000002 2020-03-09T00:25:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833219+0000", 2020-03-09T00:25:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4809999999999995e-06 2020-03-09T00:25:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833229+0000", 2020-03-09T00:25:03.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9320000000000008e-06 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833238+0000", 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0749999999999999e-06 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833243+0000", 2020-03-09T00:25:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1554000000000001e-05 2020-03-09T00:25:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:03.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833294+0000", 2020-03-09T00:25:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4169999999999999e-06 2020-03-09T00:25:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833297+0000", 2020-03-09T00:25:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00039104100000000001 2020-03-09T00:25:03.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:400 3.7s0 3.f05ab707 (undecoded) ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:03.895 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.035627998000000001, 2020-03-09T00:25:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.035791806000000002, 2020-03-09T00:25:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "delayed", 2020-03-09T00:25:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:03.896 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 400 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:03.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9441999999999999e-05 2020-03-09T00:25:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833319+0000", 2020-03-09T00:25:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.421e-06 2020-03-09T00:25:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833325+0000", 2020-03-09T00:25:03.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0309999999999998e-06 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833333+0000", 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6189999999999998e-06 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833336+0000", 2020-03-09T00:25:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5015999999999998e-05 2020-03-09T00:25:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:03.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833371+0000", 2020-03-09T00:25:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.305e-06 2020-03-09T00:25:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833372+0000", 2020-03-09T00:25:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2833999999999999e-05 2020-03-09T00:25:03.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:401 3.7s0 3.f05ab707 (undecoded) ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.035546970999999997, 2020-03-09T00:25:03.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.035748332000000001, 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "delayed", 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 401 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.904 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3272e-05 2020-03-09T00:25:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833383+0000", 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2030000000000001e-06 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833387+0000", 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0190000000000006e-06 2020-03-09T00:25:03.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833395+0000", 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3800000000000001e-06 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833397+0000", 2020-03-09T00:25:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9891000000000001e-05 2020-03-09T00:25:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833437+0000", 2020-03-09T00:25:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.564e-06 2020-03-09T00:25:03.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833438+0000", 2020-03-09T00:25:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8329000000000002e-05 2020-03-09T00:25:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:03.910 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2020-03-09T00:25:03.911 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 5 2020-03-09T00:25:03.911 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:03.912 INFO:tasks.rados.rados.0.smithi104.stdout:151: finishing write tid 1 to smithi10428818-17 2020-03-09T00:25:03.912 INFO:tasks.rados.rados.0.smithi104.stdout:151: finishing write tid 2 to smithi10428818-17 2020-03-09T00:25:03.912 INFO:tasks.rados.rados.0.smithi104.stdout:151: finishing write tid 3 to smithi10428818-17 2020-03-09T00:25:03.912 INFO:tasks.rados.rados.0.smithi104.stdout:151: finishing write tid 5 to smithi10428818-17 2020-03-09T00:25:03.913 INFO:tasks.rados.rados.0.smithi104.stdout:151: finishing write tid 6 to smithi10428818-17 2020-03-09T00:25:03.913 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 17 v 19 (ObjNum 77 snap 12 seq_num 77) dirty exists 2020-03-09T00:25:03.913 INFO:tasks.rados.rados.0.smithi104.stdout:151: left oid 17 (ObjNum 77 snap 12 seq_num 77) 2020-03-09T00:25:03.913 INFO:tasks.rados.rados.0.smithi104.stdout:150: expect (ObjNum 73 snap 11 seq_num 73) 2020-03-09T00:25:03.914 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:25:03.959 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:03.957+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:03.976 INFO:tasks.rados.rados.0.smithi104.stdout:156: expect (ObjNum 23 snap 0 seq_num 23) 2020-03-09T00:25:03.986 INFO:tasks.rados.rados.0.smithi104.stdout:153: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:25:04.014 INFO:tasks.rados.rados.0.smithi104.stdout:152: finishing write tid 1 to smithi10428818-27 2020-03-09T00:25:04.014 INFO:tasks.rados.rados.0.smithi104.stdout:152: finishing write tid 2 to smithi10428818-27 2020-03-09T00:25:04.014 INFO:tasks.rados.rados.0.smithi104.stdout:152: finishing write tid 3 to smithi10428818-27 2020-03-09T00:25:04.014 INFO:tasks.rados.rados.0.smithi104.stdout:152: finishing write tid 5 to smithi10428818-27 2020-03-09T00:25:04.015 INFO:tasks.rados.rados.0.smithi104.stdout:154: expect (ObjNum 59 snap 6 seq_num 59) 2020-03-09T00:25:04.029 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:04.029 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:04.030 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:04.030 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:04.030 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:04.033 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:04.032+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:04.038 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:25:04.050 INFO:tasks.rados.rados.0.smithi104.stdout:152: finishing write tid 6 to smithi10428818-27 2020-03-09T00:25:04.050 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 27 v 45 (ObjNum 78 snap 12 seq_num 78) dirty exists 2020-03-09T00:25:04.050 INFO:tasks.rados.rados.0.smithi104.stdout:152: left oid 27 (ObjNum 78 snap 12 seq_num 78) 2020-03-09T00:25:04.050 INFO:tasks.rados.rados.0.smithi104.stdout:155: expect (ObjNum 38 snap 0 seq_num 38) 2020-03-09T00:25:04.065 INFO:tasks.rados.rados.0.smithi104.stdout:161: expect (ObjNum 71 snap 11 seq_num 71) 2020-03-09T00:25:04.134 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:04.132+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:04.179 INFO:tasks.rados.rados.0.smithi104.stdout:148: done (13 left) 2020-03-09T00:25:04.179 INFO:tasks.rados.rados.0.smithi104.stdout:149: done (12 left) 2020-03-09T00:25:04.179 INFO:tasks.rados.rados.0.smithi104.stdout:150: done (11 left) 2020-03-09T00:25:04.179 INFO:tasks.rados.rados.0.smithi104.stdout:151: done (10 left) 2020-03-09T00:25:04.179 INFO:tasks.rados.rados.0.smithi104.stdout:152: done (9 left) 2020-03-09T00:25:04.179 INFO:tasks.rados.rados.0.smithi104.stdout:153: done (8 left) 2020-03-09T00:25:04.179 INFO:tasks.rados.rados.0.smithi104.stdout:154: done (7 left) 2020-03-09T00:25:04.180 INFO:tasks.rados.rados.0.smithi104.stdout:155: done (6 left) 2020-03-09T00:25:04.180 INFO:tasks.rados.rados.0.smithi104.stdout:156: done (5 left) 2020-03-09T00:25:04.180 INFO:tasks.rados.rados.0.smithi104.stdout:161: done (4 left) 2020-03-09T00:25:04.180 INFO:tasks.rados.rados.0.smithi104.stdout:163: delete oid 42 current snap is 12 2020-03-09T00:25:04.184 INFO:tasks.rados.rados.0.smithi104.stdout:163: done (4 left) 2020-03-09T00:25:04.184 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 20 current snap is 12 2020-03-09T00:25:04.184 INFO:tasks.rados.rados.0.smithi104.stdout:164: seq_num 82 ranges {2572288=622592,3194880=475136,3670016=294912} 2020-03-09T00:25:04.195 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:04.195 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:04.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:04.195 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:04.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.195 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:179 3.7s0 3:e7e20030:::smithi10428818-9:head [read 0~147456,checksum] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:46.811037+0000", 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.381415273000002, 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076295599999999996, 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.196 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.197 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 179 2020-03-09T00:25:04.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.197 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811037+0000", 2020-03-09T00:25:04.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811037+0000", 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1561000000000001e-05 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811108+0000", 2020-03-09T00:25:04.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1589999999999998e-06 2020-03-09T00:25:04.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811114+0000", 2020-03-09T00:25:04.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3157e-05 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811128+0000", 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6450000000000004e-06 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811132+0000", 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6229e-05 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811188+0000", 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1640000000000001e-06 2020-03-09T00:25:04.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:04.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.811197+0000", 2020-03-09T00:25:04.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039656980000000001 2020-03-09T00:25:04.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815162+0000", 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018545500000000001 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.815348+0000", 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0033184920000000001 2020-03-09T00:25:04.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:46.818666+0000", 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076295599999999996 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.205 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.5s2 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:04.205 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.827763+0000", 2020-03-09T00:25:04.205 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.364688605000001, 2020-03-09T00:25:04.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0085507429999999995, 2020-03-09T00:25:04.205 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.205 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827763+0000", 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.827763+0000", 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000278683 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828042+0000", 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4984e-05 2020-03-09T00:25:04.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828067+0000", 2020-03-09T00:25:04.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5858e-05 2020-03-09T00:25:04.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828083+0000", 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.345e-06 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828090+0000", 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000113941 2020-03-09T00:25:04.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828204+0000", 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3959e-05 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.828218+0000", 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0080567880000000005 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836275+0000", 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9184999999999998e-05 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.836314+0000", 2020-03-09T00:25:04.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0085507429999999995 2020-03-09T00:25:04.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.212 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.8s4 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:47.837641+0000", 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.354810525000001, 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086515399999999992, 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:04.213 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837641+0000", 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837641+0000", 2020-03-09T00:25:04.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000207486 2020-03-09T00:25:04.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837849+0000", 2020-03-09T00:25:04.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0214999999999999e-05 2020-03-09T00:25:04.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837869+0000", 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2768999999999999e-05 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837882+0000", 2020-03-09T00:25:04.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0450000000000003e-06 2020-03-09T00:25:04.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837889+0000", 2020-03-09T00:25:04.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0652000000000002e-05 2020-03-09T00:25:04.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837940+0000", 2020-03-09T00:25:04.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7520000000000008e-06 2020-03-09T00:25:04.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.837949+0000", 2020-03-09T00:25:04.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0082991139999999998 2020-03-09T00:25:04.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.846248+0000", 2020-03-09T00:25:04.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4506999999999999e-05 2020-03-09T00:25:04.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:47.846293+0000", 2020-03-09T00:25:04.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086515399999999992 2020-03-09T00:25:04.220 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.221 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.221 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.221 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:233 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1048576,checksum] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:25:04.221 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:51.849236+0000", 2020-03-09T00:25:04.221 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.343215468, 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007933285, 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 233 2020-03-09T00:25:04.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.223 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849236+0000", 2020-03-09T00:25:04.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849236+0000", 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6334999999999994e-05 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849303+0000", 2020-03-09T00:25:04.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9600000000000002e-06 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849307+0000", 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2362e-05 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849319+0000", 2020-03-09T00:25:04.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1509999999999997e-06 2020-03-09T00:25:04.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849323+0000", 2020-03-09T00:25:04.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000715611 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.850039+0000", 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1669999999999997e-05 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.850101+0000", 2020-03-09T00:25:04.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0070691959999999998 2020-03-09T00:25:04.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.857170+0000", 2020-03-09T00:25:04.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007933285 2020-03-09T00:25:04.229 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:234 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1048576,checksum] snapc 0=[] ondisk+read+known_if_redirected e42)", 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:51.849363+0000", 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.343088915999999, 2020-03-09T00:25:04.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0084719410000000002, 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 234 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.231 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849363+0000", 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849363+0000", 2020-03-09T00:25:04.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9310999999999997e-05 2020-03-09T00:25:04.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849402+0000", 2020-03-09T00:25:04.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4379999999999999e-06 2020-03-09T00:25:04.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849406+0000", 2020-03-09T00:25:04.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5100000000000004e-06 2020-03-09T00:25:04.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849412+0000", 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.773e-06 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.849415+0000", 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0030782209999999999 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852493+0000", 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010695400000000001 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.237 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.852600+0000", 2020-03-09T00:25:04.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0052347339999999996 2020-03-09T00:25:04.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:51.857835+0000", 2020-03-09T00:25:04.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0084719410000000002 2020-03-09T00:25:04.239 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.239 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.239 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.239 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:253 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [delete] snapc 9=[9,5,4] ondisk+write+peerstat_old+known_if_redirected e44)", 2020-03-09T00:25:04.239 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.850040+0000", 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.342412137, 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095750309999999995, 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.240 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 253 2020-03-09T00:25:04.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.241 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850040+0000", 2020-03-09T00:25:04.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850040+0000", 2020-03-09T00:25:04.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6847000000000001e-05 2020-03-09T00:25:04.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850087+0000", 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7579999999999992e-06 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850094+0000", 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6542000000000001e-05 2020-03-09T00:25:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850121+0000", 2020-03-09T00:25:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1762e-05 2020-03-09T00:25:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850133+0000", 2020-03-09T00:25:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6909000000000001e-05 2020-03-09T00:25:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850180+0000", 2020-03-09T00:25:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000112638 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.850292+0000", 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00086019000000000004 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851152+0000", 2020-03-09T00:25:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001529798 2020-03-09T00:25:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.852682+0000", 2020-03-09T00:25:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068435950000000001 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859526+0000", 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8992000000000004e-05 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859615+0000", 2020-03-09T00:25:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095750309999999995 2020-03-09T00:25:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s2 49/32 ECSubWrite(tid=77, reqid=client.4751.0:297, at_version=49'29, trim_to=0'0, roll_forward_to=49'27))", 2020-03-09T00:25:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.828919+0000", 2020-03-09T00:25:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3635328429999998, 2020-03-09T00:25:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0084093029999999999, 2020-03-09T00:25:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:04.250 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.828919+0000", 2020-03-09T00:25:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.828919+0000", 2020-03-09T00:25:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000571193 2020-03-09T00:25:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.829490+0000", 2020-03-09T00:25:04.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5803999999999999e-05 2020-03-09T00:25:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.829516+0000", 2020-03-09T00:25:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4969e-05 2020-03-09T00:25:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.829531+0000", 2020-03-09T00:25:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1149999999999996e-06 2020-03-09T00:25:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.829538+0000", 2020-03-09T00:25:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3228e-05 2020-03-09T00:25:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.829581+0000", 2020-03-09T00:25:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3180000000000007e-06 2020-03-09T00:25:04.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.829590+0000", 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0076719070000000004 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.256 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837262+0000", 2020-03-09T00:25:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6768999999999997e-05 2020-03-09T00:25:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837328+0000", 2020-03-09T00:25:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0084093029999999999 2020-03-09T00:25:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/32 ECSubWrite(tid=77, reqid=client.4751.0:301, at_version=49'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.831217+0000", 2020-03-09T00:25:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.361234617, 2020-03-09T00:25:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014280915, 2020-03-09T00:25:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831217+0000", 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831217+0000", 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00086706800000000003 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.832084+0000", 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8614000000000001e-05 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.832113+0000", 2020-03-09T00:25:04.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8689e-05 2020-03-09T00:25:04.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.832132+0000", 2020-03-09T00:25:04.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8199999999999997e-06 2020-03-09T00:25:04.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.832140+0000", 2020-03-09T00:25:04.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013448850000000001 2020-03-09T00:25:04.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.833484+0000", 2020-03-09T00:25:04.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5074e-05 2020-03-09T00:25:04.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.833499+0000", 2020-03-09T00:25:04.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011952076000000001 2020-03-09T00:25:04.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.266 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845452+0000", 2020-03-09T00:25:04.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6689e-05 2020-03-09T00:25:04.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.266 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.266 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845498+0000", 2020-03-09T00:25:04.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014280915 2020-03-09T00:25:04.267 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.267 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.267 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.267 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.267 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/32 ECSubWrite(tid=78, reqid=client.4751.0:302, at_version=49'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:04.267 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.835523+0000", 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3569293250000003, 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010127255999999999, 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.268 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835523+0000", 2020-03-09T00:25:04.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835523+0000", 2020-03-09T00:25:04.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00019369600000000001 2020-03-09T00:25:04.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835716+0000", 2020-03-09T00:25:04.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0733e-05 2020-03-09T00:25:04.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835737+0000", 2020-03-09T00:25:04.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7965000000000002e-05 2020-03-09T00:25:04.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835755+0000", 2020-03-09T00:25:04.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.5989999999999996e-06 2020-03-09T00:25:04.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.274 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835763+0000", 2020-03-09T00:25:04.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056860790000000001 2020-03-09T00:25:04.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.275 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.841449+0000", 2020-03-09T00:25:04.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1802e-05 2020-03-09T00:25:04.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.276 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.841460+0000", 2020-03-09T00:25:04.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041435270000000001 2020-03-09T00:25:04.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.277 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845604+0000", 2020-03-09T00:25:04.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5855000000000001e-05 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845650+0000", 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010127255999999999 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.278 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.279 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:311 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~770048] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:04.279 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:04.279 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.4087198889999999, 2020-03-09T00:25:04.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438, 2020-03-09T00:25:04.279 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.280 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:04.280 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.280 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.280 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.280 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 311 2020-03-09T00:25:04.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.280 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:04.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.282 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:04.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014001199999999999 2020-03-09T00:25:04.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.282 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.282 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785132+0000", 2020-03-09T00:25:04.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1973999999999999e-05 2020-03-09T00:25:04.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.283 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.283 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785144+0000", 2020-03-09T00:25:04.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1815e-05 2020-03-09T00:25:04.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.284 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785156+0000", 2020-03-09T00:25:04.284 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2449999999999998e-06 2020-03-09T00:25:04.284 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.284 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785161+0000", 2020-03-09T00:25:04.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00063616300000000003 2020-03-09T00:25:04.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785797+0000", 2020-03-09T00:25:04.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00065134999999999998 2020-03-09T00:25:04.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.286 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.286 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.786449+0000", 2020-03-09T00:25:04.286 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00188737 2020-03-09T00:25:04.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788336+0000", 2020-03-09T00:25:04.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010887625999999999 2020-03-09T00:25:04.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799224+0000", 2020-03-09T00:25:04.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037060560000000001 2020-03-09T00:25:04.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:04.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.802930+0000", 2020-03-09T00:25:04.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000460719 2020-03-09T00:25:04.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803390+0000", 2020-03-09T00:25:04.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438 2020-03-09T00:25:04.289 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.289 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.290 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.290 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:312 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~376832] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:04.290 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:04.290 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.4072237409999999, 2020-03-09T00:25:04.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001, 2020-03-09T00:25:04.291 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.291 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:04.291 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.291 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.291 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.291 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 312 2020-03-09T00:25:04.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.292 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:04.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:04.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026595500000000001 2020-03-09T00:25:04.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785494+0000", 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3902e-05 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785508+0000", 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.417e-05 2020-03-09T00:25:04.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.295 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785522+0000", 2020-03-09T00:25:04.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.99e-06 2020-03-09T00:25:04.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.296 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785529+0000", 2020-03-09T00:25:04.296 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083854199999999993 2020-03-09T00:25:04.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.296 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.296 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793915+0000", 2020-03-09T00:25:04.296 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024809000000000002 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.794163+0000", 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001122184 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795285+0000", 2020-03-09T00:25:04.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004034358 2020-03-09T00:25:04.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799319+0000", 2020-03-09T00:25:04.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042626369999999997 2020-03-09T00:25:04.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:04.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803582+0000", 2020-03-09T00:25:04.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00042245199999999997 2020-03-09T00:25:04.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804004+0000", 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.301 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:313 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:04.301 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:04.301 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.40686818, 2020-03-09T00:25:04.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001, 2020-03-09T00:25:04.301 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.301 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 313 2020-03-09T00:25:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:04.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:04.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7169000000000004e-05 2020-03-09T00:25:04.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785681+0000", 2020-03-09T00:25:04.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8849999999999999e-06 2020-03-09T00:25:04.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.305 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785689+0000", 2020-03-09T00:25:04.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5290000000000001e-05 2020-03-09T00:25:04.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785704+0000", 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5759999999999996e-06 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785710+0000", 2020-03-09T00:25:04.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012205476 2020-03-09T00:25:04.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.307 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.307 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.307 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.797915+0000", 2020-03-09T00:25:04.307 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023467400000000001 2020-03-09T00:25:04.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.308 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.308 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.798150+0000", 2020-03-09T00:25:04.308 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00085795900000000004 2020-03-09T00:25:04.308 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.308 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.309 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799008+0000", 2020-03-09T00:25:04.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00050973499999999998 2020-03-09T00:25:04.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799518+0000", 2020-03-09T00:25:04.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046306960000000001 2020-03-09T00:25:04.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:04.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804148+0000", 2020-03-09T00:25:04.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00032100500000000002 2020-03-09T00:25:04.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804469+0000", 2020-03-09T00:25:04.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001 2020-03-09T00:25:04.311 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.311 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.312 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.312 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:314 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:04.312 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:04.312 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.4066854, 2020-03-09T00:25:04.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998, 2020-03-09T00:25:04.313 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.313 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.313 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.313 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.313 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.313 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 314 2020-03-09T00:25:04.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.314 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.314 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:04.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.315 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.315 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:04.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9129999999999999e-05 2020-03-09T00:25:04.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.315 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785816+0000", 2020-03-09T00:25:04.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1280000000000002e-06 2020-03-09T00:25:04.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785823+0000", 2020-03-09T00:25:04.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.24e-05 2020-03-09T00:25:04.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.317 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785835+0000", 2020-03-09T00:25:04.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1800000000000004e-06 2020-03-09T00:25:04.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785840+0000", 2020-03-09T00:25:04.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014080127 2020-03-09T00:25:04.318 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799920+0000", 2020-03-09T00:25:04.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023000599999999999 2020-03-09T00:25:04.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:04.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.800150+0000", 2020-03-09T00:25:04.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048975440000000002 2020-03-09T00:25:04.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.320 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.320 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805048+0000", 2020-03-09T00:25:04.320 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000221747 2020-03-09T00:25:04.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805270+0000", 2020-03-09T00:25:04.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037657239999999998 2020-03-09T00:25:04.321 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.321 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.321 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.321 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.809036+0000", 2020-03-09T00:25:04.322 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998 2020-03-09T00:25:04.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.322 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.322 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.322 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:351 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1048576,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e53)", 2020-03-09T00:25:04.323 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.831208+0000", 2020-03-09T00:25:04.323 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3612440690000001, 2020-03-09T00:25:04.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0078443030000000004, 2020-03-09T00:25:04.323 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.323 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.323 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.323 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.324 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.324 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 351 2020-03-09T00:25:04.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.324 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831208+0000", 2020-03-09T00:25:04.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831208+0000", 2020-03-09T00:25:04.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1712000000000006e-05 2020-03-09T00:25:04.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831290+0000", 2020-03-09T00:25:04.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6189999999999999e-06 2020-03-09T00:25:04.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831295+0000", 2020-03-09T00:25:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0721e-05 2020-03-09T00:25:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831306+0000", 2020-03-09T00:25:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2756e-05 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831319+0000", 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9251e-05 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831368+0000", 2020-03-09T00:25:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00020493700000000001 2020-03-09T00:25:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831573+0000", 2020-03-09T00:25:04.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074793070000000001 2020-03-09T00:25:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.839052+0000", 2020-03-09T00:25:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0078443030000000004 2020-03-09T00:25:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.330 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:352 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1048576,checksum] snapc 0=[] ondisk+read+known_if_redirected e53)", 2020-03-09T00:25:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.831377+0000", 2020-03-09T00:25:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.361074602, 2020-03-09T00:25:04.331 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090925329999999999, 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 352 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.332 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831377+0000", 2020-03-09T00:25:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.333 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831377+0000", 2020-03-09T00:25:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010362899999999999 2020-03-09T00:25:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831481+0000", 2020-03-09T00:25:04.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.5809999999999999e-06 2020-03-09T00:25:04.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.335 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.335 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.335 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831489+0000", 2020-03-09T00:25:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3711e-05 2020-03-09T00:25:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831502+0000", 2020-03-09T00:25:04.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3999999999999997e-06 2020-03-09T00:25:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831509+0000", 2020-03-09T00:25:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00062636699999999996 2020-03-09T00:25:04.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.832135+0000", 2020-03-09T00:25:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013581699999999999 2020-03-09T00:25:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.832271+0000", 2020-03-09T00:25:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081990280000000006 2020-03-09T00:25:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.840470+0000", 2020-03-09T00:25:04.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090925329999999999 2020-03-09T00:25:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:353 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1048576,checksum] snapc 0=[] ondisk+read+known_if_redirected e53)", 2020-03-09T00:25:04.340 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.831652+0000", 2020-03-09T00:25:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.360799603, 2020-03-09T00:25:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0099000790000000009, 2020-03-09T00:25:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.341 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 353 2020-03-09T00:25:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831652+0000", 2020-03-09T00:25:04.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.345 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.345 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.345 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831652+0000", 2020-03-09T00:25:04.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7905999999999997e-05 2020-03-09T00:25:04.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.346 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831720+0000", 2020-03-09T00:25:04.346 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6530000000000002e-06 2020-03-09T00:25:04.347 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.347 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831725+0000", 2020-03-09T00:25:04.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1769999999999998e-06 2020-03-09T00:25:04.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.348 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.348 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831733+0000", 2020-03-09T00:25:04.349 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1869999999999999e-06 2020-03-09T00:25:04.349 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.349 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.349 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.831737+0000", 2020-03-09T00:25:04.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00155127 2020-03-09T00:25:04.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.350 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.833289+0000", 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.3751000000000006e-05 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.833362+0000", 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081901349999999994 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.352 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.352 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.841552+0000", 2020-03-09T00:25:04.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0099000790000000009 2020-03-09T00:25:04.352 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.352 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.352 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.3409303610000001, 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001, 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.353 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:04.354 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.354 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.354 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:04.354 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.354 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0030360970000000002 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854558+0000", 2020-03-09T00:25:04.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8904e-05 2020-03-09T00:25:04.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.356 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.356 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854587+0000", 2020-03-09T00:25:04.356 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5423000000000001e-05 2020-03-09T00:25:04.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854602+0000", 2020-03-09T00:25:04.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9139999999999997e-06 2020-03-09T00:25:04.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.357 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854611+0000", 2020-03-09T00:25:04.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2179000000000002e-05 2020-03-09T00:25:04.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.358 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.358 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854673+0000", 2020-03-09T00:25:04.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6330000000000001e-05 2020-03-09T00:25:04.359 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.359 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854689+0000", 2020-03-09T00:25:04.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019248372999999999 2020-03-09T00:25:04.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.360 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873938+0000", 2020-03-09T00:25:04.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3464000000000003e-05 2020-03-09T00:25:04.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.361 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873981+0000", 2020-03-09T00:25:04.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001 2020-03-09T00:25:04.361 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.361 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.362 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.362 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.bs6 53/32 ECSubWrite(tid=57, reqid=client.4751.0:362, at_version=53'28, trim_to=0'0, roll_forward_to=52'26))", 2020-03-09T00:25:04.362 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.894104+0000", 2020-03-09T00:25:04.362 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.2983481400000001, 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0085930520000000003, 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894104+0000", 2020-03-09T00:25:04.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894104+0000", 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.5118999999999995e-05 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894199+0000", 2020-03-09T00:25:04.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6733e-05 2020-03-09T00:25:04.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894226+0000", 2020-03-09T00:25:04.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7482999999999998e-05 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894243+0000", 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4889999999999995e-06 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894252+0000", 2020-03-09T00:25:04.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4318999999999999e-05 2020-03-09T00:25:04.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.368 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.368 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894306+0000", 2020-03-09T00:25:04.368 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3057999999999999e-05 2020-03-09T00:25:04.368 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:04.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.894319+0000", 2020-03-09T00:25:04.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0082329880000000001 2020-03-09T00:25:04.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:04.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.902552+0000", 2020-03-09T00:25:04.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014486300000000001 2020-03-09T00:25:04.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.902697+0000", 2020-03-09T00:25:04.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0085930520000000003 2020-03-09T00:25:04.371 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.371 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.371 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:386 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1048576,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.817807+0000", 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.37464449100000002, 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0075590309999999999, 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.372 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.373 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.373 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 386 2020-03-09T00:25:04.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.373 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817807+0000", 2020-03-09T00:25:04.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817807+0000", 2020-03-09T00:25:04.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8584000000000002e-05 2020-03-09T00:25:04.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817846+0000", 2020-03-09T00:25:04.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3850000000000002e-06 2020-03-09T00:25:04.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817849+0000", 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0079999999999996e-06 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817856+0000", 2020-03-09T00:25:04.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6569999999999999e-06 2020-03-09T00:25:04.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.377 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.377 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817860+0000", 2020-03-09T00:25:04.377 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00067058700000000005 2020-03-09T00:25:04.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.378 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.378 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818531+0000", 2020-03-09T00:25:04.378 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4143999999999999e-05 2020-03-09T00:25:04.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.379 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818625+0000", 2020-03-09T00:25:04.379 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067416660000000003 2020-03-09T00:25:04.379 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.379 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.379 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.825366+0000", 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0075590309999999999 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:387 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1048576,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:04.380 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.817912+0000", 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.374540023, 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095343219999999996, 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:04.381 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 387 2020-03-09T00:25:04.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.382 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:04.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.382 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:04.382 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817912+0000", 2020-03-09T00:25:04.382 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:04.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817912+0000", 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7327e-05 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:04.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817989+0000", 2020-03-09T00:25:04.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9500000000000004e-06 2020-03-09T00:25:04.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:04.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.817996+0000", 2020-03-09T00:25:04.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3069e-05 2020-03-09T00:25:04.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:04.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818009+0000", 2020-03-09T00:25:04.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4620000000000004e-06 2020-03-09T00:25:04.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:04.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818015+0000", 2020-03-09T00:25:04.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0017506640000000001 2020-03-09T00:25:04.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.386 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:04.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819765+0000", 2020-03-09T00:25:04.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017303699999999999 2020-03-09T00:25:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819938+0000", 2020-03-09T00:25:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0075078130000000003 2020-03-09T00:25:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827446+0000", 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095343219999999996 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:04.388 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:04.389 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:04.392 INFO:tasks.rados.rados.0.smithi104.stdout:164: writing smithi10428818-20 from 2572288 to 3194880 tid 1 2020-03-09T00:25:04.393 INFO:tasks.rados.rados.0.smithi104.stdout:164: writing smithi10428818-20 from 3194880 to 3670016 tid 2 2020-03-09T00:25:04.393 INFO:tasks.rados.rados.0.smithi104.stdout:164: writing smithi10428818-20 from 3670016 to 3964928 tid 3 2020-03-09T00:25:04.393 INFO:tasks.rados.rados.0.smithi104.stdout:165: rollback oid 3 current snap is 12 2020-03-09T00:25:04.393 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 3 to 9 2020-03-09T00:25:04.393 INFO:tasks.rados.rados.0.smithi104.stdout:166: rmattr oid 32 current snap is 12 2020-03-09T00:25:04.393 INFO:tasks.rados.rados.0.smithi104.stdout:166: done (6 left) 2020-03-09T00:25:04.393 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 33 current snap is 12 2020-03-09T00:25:04.394 INFO:tasks.rados.rados.0.smithi104.stdout:167: seq_num 83 ranges {0=638976,638976=622592,1261568=147456} 2020-03-09T00:25:04.394 INFO:tasks.rados.rados.0.smithi104.stdout:167: writing smithi10428818-33 from 0 to 638976 tid 1 2020-03-09T00:25:04.394 INFO:tasks.rados.rados.0.smithi104.stdout:167: writing smithi10428818-33 from 638976 to 1261568 tid 2 2020-03-09T00:25:04.394 INFO:tasks.rados.rados.0.smithi104.stdout:167: writing smithi10428818-33 from 1261568 to 1409024 tid 3 2020-03-09T00:25:04.394 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 35 current snap is 12 2020-03-09T00:25:04.394 INFO:tasks.rados.rados.0.smithi104.stdout:168: seq_num 84 ranges {573440=655360,1228800=638976,1867776=131072} 2020-03-09T00:25:04.395 INFO:tasks.rados.rados.0.smithi104.stdout:168: writing smithi10428818-35 from 573440 to 1228800 tid 1 2020-03-09T00:25:04.395 INFO:tasks.rados.rados.0.smithi104.stdout:168: writing smithi10428818-35 from 1228800 to 1867776 tid 2 2020-03-09T00:25:04.395 INFO:tasks.rados.rados.0.smithi104.stdout:168: writing smithi10428818-35 from 1867776 to 1998848 tid 3 2020-03-09T00:25:04.395 INFO:tasks.rados.rados.0.smithi104.stdout:169: snap_create 2020-03-09T00:25:04.395 INFO:tasks.rados.rados.0.smithi104.stdout:164: finishing write tid 1 to smithi10428818-20 2020-03-09T00:25:04.396 INFO:tasks.rados.rados.0.smithi104.stdout:164: finishing write tid 2 to smithi10428818-20 2020-03-09T00:25:04.396 INFO:tasks.rados.rados.0.smithi104.stdout:164: finishing write tid 3 to smithi10428818-20 2020-03-09T00:25:04.396 INFO:tasks.rados.rados.0.smithi104.stdout:164: finishing write tid 5 to smithi10428818-20 2020-03-09T00:25:04.396 INFO:tasks.rados.rados.0.smithi104.stdout:164: finishing write tid 6 to smithi10428818-20 2020-03-09T00:25:04.397 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 20 v 47 (ObjNum 82 snap 12 seq_num 82) dirty exists 2020-03-09T00:25:04.397 INFO:tasks.rados.rados.0.smithi104.stdout:164: left oid 20 (ObjNum 82 snap 12 seq_num 82) 2020-03-09T00:25:04.397 INFO:tasks.rados.rados.0.smithi104.stdout:167: finishing write tid 1 to smithi10428818-33 2020-03-09T00:25:04.397 INFO:tasks.rados.rados.0.smithi104.stdout:167: finishing write tid 2 to smithi10428818-33 2020-03-09T00:25:04.397 INFO:tasks.rados.rados.0.smithi104.stdout:167: finishing write tid 3 to smithi10428818-33 2020-03-09T00:25:04.398 INFO:tasks.rados.rados.0.smithi104.stdout:167: finishing write tid 5 to smithi10428818-33 2020-03-09T00:25:04.398 INFO:tasks.rados.rados.0.smithi104.stdout:168: finishing write tid 1 to smithi10428818-35 2020-03-09T00:25:04.398 INFO:tasks.rados.rados.0.smithi104.stdout:168: finishing write tid 2 to smithi10428818-35 2020-03-09T00:25:04.398 INFO:tasks.rados.rados.0.smithi104.stdout:167: finishing write tid 6 to smithi10428818-33 2020-03-09T00:25:04.399 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 33 v 48 (ObjNum 83 snap 12 seq_num 83) dirty exists 2020-03-09T00:25:04.399 INFO:tasks.rados.rados.0.smithi104.stdout:167: left oid 33 (ObjNum 83 snap 12 seq_num 83) 2020-03-09T00:25:04.399 INFO:tasks.rados.rados.0.smithi104.stdout:168: finishing write tid 3 to smithi10428818-35 2020-03-09T00:25:04.399 INFO:tasks.rados.rados.0.smithi104.stdout:168: finishing write tid 5 to smithi10428818-35 2020-03-09T00:25:04.399 INFO:tasks.rados.rados.0.smithi104.stdout:168: finishing write tid 6 to smithi10428818-35 2020-03-09T00:25:04.400 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 35 v 21 (ObjNum 84 snap 12 seq_num 84) dirty exists 2020-03-09T00:25:04.400 INFO:tasks.rados.rados.0.smithi104.stdout:168: left oid 35 (ObjNum 84 snap 12 seq_num 84) 2020-03-09T00:25:04.400 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:04.388+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:04.401 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:25:04.491 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:04.490+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:04.537 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:04.537 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:04.537 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:04.537 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:04.546 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:25:04.585 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:04.592 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:04.591+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:04.669 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:04.669 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:04.669 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:04.669 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:04.670 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:04.677 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:25:04.693 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:04.692+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:04.719 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:04.739 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:04.759 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:04.779 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:04.794 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:04.793+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:04.795 INFO:tasks.rados.rados.0.smithi104.stdout:164: done (8 left) 2020-03-09T00:25:04.796 INFO:tasks.rados.rados.0.smithi104.stdout:167: done (7 left) 2020-03-09T00:25:04.796 INFO:tasks.rados.rados.0.smithi104.stdout:168: done (6 left) 2020-03-09T00:25:04.796 INFO:tasks.rados.rados.0.smithi104.stdout:169: done (5 left) 2020-03-09T00:25:04.796 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 13 current snap is 13 2020-03-09T00:25:04.797 INFO:tasks.rados.rados.0.smithi104.stdout:170: seq_num 85 ranges {212992=671744,884736=770048} 2020-03-09T00:25:04.800 INFO:tasks.rados.rados.0.smithi104.stdout:170: writing smithi10428818-13 from 212992 to 884736 tid 1 2020-03-09T00:25:04.805 INFO:tasks.rados.rados.0.smithi104.stdout:170: writing smithi10428818-13 from 884736 to 1654784 tid 2 2020-03-09T00:25:04.806 INFO:tasks.rados.rados.0.smithi104.stdout:171: snap_create 2020-03-09T00:25:04.817 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:04.823 INFO:tasks.rados.rados.0.smithi104.stdout:165: finishing rollback tid 1 to smithi10428818-3 2020-03-09T00:25:04.823 INFO:tasks.rados.rados.0.smithi104.stdout:165: finishing rollback tid 2 to smithi10428818-3 2020-03-09T00:25:04.823 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 3 v 23 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-03-09T00:25:04.824 INFO:tasks.rados.rados.0.smithi104.stdout:162: finishing copy_from to smithi10428818-49 2020-03-09T00:25:04.824 INFO:tasks.rados.rados.0.smithi104.stdout:162: got expected ENOENT (src dne) 2020-03-09T00:25:04.826 INFO:tasks.rados.rados.0.smithi104.stdout:162: finishing copy_from racing read to smithi10428818-49 2020-03-09T00:25:04.828 INFO:tasks.rados.rados.0.smithi104.stdout:157: finishing write tid 1 to smithi10428818-23 2020-03-09T00:25:04.828 INFO:tasks.rados.rados.0.smithi104.stdout:157: finishing write tid 2 to smithi10428818-23 2020-03-09T00:25:04.829 INFO:tasks.rados.rados.0.smithi104.stdout:157: finishing write tid 3 to smithi10428818-23 2020-03-09T00:25:04.829 INFO:tasks.rados.rados.0.smithi104.stdout:157: finishing write tid 5 to smithi10428818-23 2020-03-09T00:25:04.832 INFO:tasks.rados.rados.0.smithi104.stdout:157: finishing write tid 6 to smithi10428818-23 2020-03-09T00:25:04.832 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 23 v 20 (ObjNum 79 snap 12 seq_num 79) dirty exists 2020-03-09T00:25:04.832 INFO:tasks.rados.rados.0.smithi104.stdout:157: left oid 23 (ObjNum 79 snap 12 seq_num 79) 2020-03-09T00:25:04.835 INFO:tasks.rados.rados.0.smithi104.stdout:170: finishing write tid 1 to smithi10428818-13 2020-03-09T00:25:04.836 INFO:tasks.rados.rados.0.smithi104.stdout:159: finishing write tid 1 to smithi10428818-29 2020-03-09T00:25:04.837 INFO:tasks.rados.rados.0.smithi104.stdout:159: finishing write tid 2 to smithi10428818-29 2020-03-09T00:25:04.838 INFO:tasks.rados.rados.0.smithi104.stdout:170: finishing write tid 2 to smithi10428818-13 2020-03-09T00:25:04.839 INFO:tasks.rados.rados.0.smithi104.stdout:170: finishing write tid 4 to smithi10428818-13 2020-03-09T00:25:04.842 INFO:tasks.rados.rados.0.smithi104.stdout:159: finishing write tid 3 to smithi10428818-29 2020-03-09T00:25:04.843 INFO:tasks.rados.rados.0.smithi104.stdout:159: finishing write tid 5 to smithi10428818-29 2020-03-09T00:25:04.843 INFO:tasks.rados.rados.0.smithi104.stdout:170: finishing write tid 5 to smithi10428818-13 2020-03-09T00:25:04.843 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 13 v 51 (ObjNum 85 snap 13 seq_num 85) dirty exists 2020-03-09T00:25:04.843 INFO:tasks.rados.rados.0.smithi104.stdout:170: left oid 13 (ObjNum 85 snap 13 seq_num 85) 2020-03-09T00:25:04.844 INFO:tasks.rados.rados.0.smithi104.stdout:160: finishing copy_from to smithi10428818-19 2020-03-09T00:25:04.845 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 19 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-03-09T00:25:04.845 INFO:tasks.rados.rados.0.smithi104.stdout:160: finishing copy_from racing read to smithi10428818-19 2020-03-09T00:25:04.845 INFO:tasks.rados.rados.0.smithi104.stdout:159: finishing write tid 6 to smithi10428818-29 2020-03-09T00:25:04.846 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 29 v 29 (ObjNum 80 snap 12 seq_num 80) dirty exists 2020-03-09T00:25:04.846 INFO:tasks.rados.rados.0.smithi104.stdout:159: left oid 29 (ObjNum 80 snap 12 seq_num 80) 2020-03-09T00:25:04.854 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:04.855 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:04.855 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:04.864 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:25:04.876 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:25:04.895 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:04.894+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:04.899 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:04.919 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:04.940 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:04.960 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:04.981 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:04.986 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:04.986 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:05.003 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:05.003 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:25:05.004 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:05.001+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:05.022 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:05.042 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:05.042+0000 7f19ea27f700 0 -- 172.21.15.148:0/960292184 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7f19e4189f00 msgr2=0x7f19e418c310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:05.084 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:05.104 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:05.102+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:05.150 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:05.150 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:05.158 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:25:05.205 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:05.204+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:05.274 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:05.274 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:05.283 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:25:05.326 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:05.325+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:05.402 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:05.402 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:05.407 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:05.406+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:05.411 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:25:05.508 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:05.506+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:05.565 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:05.565 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:05.574 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:25:05.608 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:05.607+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:05.690 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:05.690 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:05.698 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:25:05.743 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:05.742+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:05.798 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 14, 15] out_osds: [13, 5] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:25:05.798 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:25:05.799 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump -f json-pretty 2020-03-09T00:25:05.801 INFO:tasks.rados.rados.0.smithi104.stdout:157: done (6 left) 2020-03-09T00:25:05.801 INFO:tasks.rados.rados.0.smithi104.stdout:159: done (5 left) 2020-03-09T00:25:05.801 INFO:tasks.rados.rados.0.smithi104.stdout:160: done (4 left) 2020-03-09T00:25:05.802 INFO:tasks.rados.rados.0.smithi104.stdout:162: done (3 left) 2020-03-09T00:25:05.802 INFO:tasks.rados.rados.0.smithi104.stdout:165: done (2 left) 2020-03-09T00:25:05.802 INFO:tasks.rados.rados.0.smithi104.stdout:170: done (1 left) 2020-03-09T00:25:05.802 INFO:tasks.rados.rados.0.smithi104.stdout:171: done (0 left) 2020-03-09T00:25:05.802 INFO:tasks.rados.rados.0.smithi104.stdout:172: read oid 36 snap -1 2020-03-09T00:25:05.802 INFO:tasks.rados.rados.0.smithi104.stdout:172: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:25:05.802 INFO:tasks.rados.rados.0.smithi104.stdout:173: read oid 24 snap -1 2020-03-09T00:25:05.803 INFO:tasks.rados.rados.0.smithi104.stdout:173: expect (ObjNum 23 snap 0 seq_num 23) 2020-03-09T00:25:05.803 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 7 current snap is 14 2020-03-09T00:25:05.803 INFO:tasks.rados.rados.0.smithi104.stdout:174: seq_num 86 ranges {0=688128,688128=540672,1228800=229376} 2020-03-09T00:25:05.805 INFO:tasks.rados.rados.0.smithi104.stdout:174: writing smithi10428818-7 from 0 to 688128 tid 1 2020-03-09T00:25:05.808 INFO:tasks.rados.rados.0.smithi104.stdout:174: writing smithi10428818-7 from 688128 to 1228800 tid 2 2020-03-09T00:25:05.810 INFO:tasks.rados.rados.0.smithi104.stdout:174: writing smithi10428818-7 from 1228800 to 1458176 tid 3 2020-03-09T00:25:05.810 INFO:tasks.rados.rados.0.smithi104.stdout:175: copy_from oid 49 from oid 44 current snap is 14 2020-03-09T00:25:05.810 INFO:tasks.rados.rados.0.smithi104.stdout:173: expect (ObjNum 23 snap 0 seq_num 23) 2020-03-09T00:25:05.820 INFO:tasks.rados.rados.0.smithi104.stdout:172: expect (ObjNum 11 snap 0 seq_num 11) 2020-03-09T00:25:05.826 INFO:tasks.rados.rados.0.smithi104.stdout:174: finishing write tid 1 to smithi10428818-7 2020-03-09T00:25:05.826 INFO:tasks.rados.rados.0.smithi104.stdout:174: finishing write tid 2 to smithi10428818-7 2020-03-09T00:25:05.826 INFO:tasks.rados.rados.0.smithi104.stdout:172: done (3 left) 2020-03-09T00:25:05.826 INFO:tasks.rados.rados.0.smithi104.stdout:173: done (2 left) 2020-03-09T00:25:05.826 INFO:tasks.rados.rados.0.smithi104.stdout:176: read oid 35 snap -1 2020-03-09T00:25:05.826 INFO:tasks.rados.rados.0.smithi104.stdout:176: expect (ObjNum 84 snap 12 seq_num 84) 2020-03-09T00:25:05.827 INFO:tasks.rados.rados.0.smithi104.stdout:177: snap_remove snap 8 2020-03-09T00:25:05.828 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-03-09T00:25:05.839 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:05.839 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:05.846 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:05.845+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:05.848 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:25:05.911 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:05.909+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:06.006 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:06.006 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:06.012 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:06.010+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:06.014 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:25:06.085 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:25:06.085 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:06.085 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 56, 2020-03-09T00:25:06.086 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "036b76b6-511e-4ec6-9669-0a47550bd261", 2020-03-09T00:25:06.086 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2020-03-09T00:24:06.736934+0000", 2020-03-09T00:25:06.086 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2020-03-09T00:25:05.792024+0000", 2020-03-09T00:25:06.086 INFO:teuthology.orchestra.run.smithi148.stdout: "last_up_change": "2020-03-09T00:24:20.875900+0000", 2020-03-09T00:25:06.086 INFO:teuthology.orchestra.run.smithi148.stdout: "last_in_change": "2020-03-09T00:25:00.770574+0000", 2020-03-09T00:25:06.086 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-09T00:25:06.087 INFO:teuthology.orchestra.run.smithi148.stdout: "flags_num": 5801984, 2020-03-09T00:25:06.087 INFO:teuthology.orchestra.run.smithi148.stdout: "flags_set": [ 2020-03-09T00:25:06.087 INFO:teuthology.orchestra.run.smithi148.stdout: "noscrub", 2020-03-09T00:25:06.087 INFO:teuthology.orchestra.run.smithi148.stdout: "pglog_hardlimit", 2020-03-09T00:25:06.087 INFO:teuthology.orchestra.run.smithi148.stdout: "purged_snapdirs", 2020-03-09T00:25:06.087 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_deletes", 2020-03-09T00:25:06.088 INFO:teuthology.orchestra.run.smithi148.stdout: "sortbitwise" 2020-03-09T00:25:06.088 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2020-03-09T00:25:06.088 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_version": 6, 2020-03-09T00:25:06.088 INFO:teuthology.orchestra.run.smithi148.stdout: "full_ratio": 0.94999998807907104, 2020-03-09T00:25:06.088 INFO:teuthology.orchestra.run.smithi148.stdout: "backfillfull_ratio": 0.89999997615814209, 2020-03-09T00:25:06.088 INFO:teuthology.orchestra.run.smithi148.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-09T00:25:06.089 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_snapshot": "", 2020-03-09T00:25:06.089 INFO:teuthology.orchestra.run.smithi148.stdout: "pool_max": 3, 2020-03-09T00:25:06.089 INFO:teuthology.orchestra.run.smithi148.stdout: "max_osd": 16, 2020-03-09T00:25:06.089 INFO:teuthology.orchestra.run.smithi148.stdout: "require_min_compat_client": "jewel", 2020-03-09T00:25:06.089 INFO:teuthology.orchestra.run.smithi148.stdout: "min_compat_client": "jewel", 2020-03-09T00:25:06.089 INFO:teuthology.orchestra.run.smithi148.stdout: "require_osd_release": "octopus", 2020-03-09T00:25:06.089 INFO:teuthology.orchestra.run.smithi148.stdout: "pools": [ 2020-03-09T00:25:06.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.090 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 1, 2020-03-09T00:25:06.090 INFO:teuthology.orchestra.run.smithi148.stdout: "pool_name": "device_health_metrics", 2020-03-09T00:25:06.090 INFO:teuthology.orchestra.run.smithi148.stdout: "create_time": "2020-03-09T00:24:09.667386+0000", 2020-03-09T00:25:06.090 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": 1, 2020-03-09T00:25:06.090 INFO:teuthology.orchestra.run.smithi148.stdout: "flags_names": "hashpspool", 2020-03-09T00:25:06.090 INFO:teuthology.orchestra.run.smithi148.stdout: "type": 1, 2020-03-09T00:25:06.091 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 2, 2020-03-09T00:25:06.091 INFO:teuthology.orchestra.run.smithi148.stdout: "min_size": 1, 2020-03-09T00:25:06.091 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_rule": 0, 2020-03-09T00:25:06.091 INFO:teuthology.orchestra.run.smithi148.stdout: "object_hash": 2, 2020-03-09T00:25:06.091 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_autoscale_mode": "off", 2020-03-09T00:25:06.091 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num": 1, 2020-03-09T00:25:06.091 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_placement_num": 1, 2020-03-09T00:25:06.092 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_placement_num_target": 1, 2020-03-09T00:25:06.092 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num_target": 1, 2020-03-09T00:25:06.092 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num_pending": 1, 2020-03-09T00:25:06.092 INFO:teuthology.orchestra.run.smithi148.stdout: "last_pg_merge_meta": { 2020-03-09T00:25:06.092 INFO:teuthology.orchestra.run.smithi148.stdout: "source_pgid": "0.0", 2020-03-09T00:25:06.092 INFO:teuthology.orchestra.run.smithi148.stdout: "ready_epoch": 0, 2020-03-09T00:25:06.092 INFO:teuthology.orchestra.run.smithi148.stdout: "last_epoch_started": 0, 2020-03-09T00:25:06.093 INFO:teuthology.orchestra.run.smithi148.stdout: "last_epoch_clean": 0, 2020-03-09T00:25:06.093 INFO:teuthology.orchestra.run.smithi148.stdout: "source_version": "0'0", 2020-03-09T00:25:06.093 INFO:teuthology.orchestra.run.smithi148.stdout: "target_version": "0'0" 2020-03-09T00:25:06.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.093 INFO:teuthology.orchestra.run.smithi148.stdout: "last_change": "26", 2020-03-09T00:25:06.093 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend": "0", 2020-03-09T00:25:06.093 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend_preluminous": "0", 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "auid": 0, 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_mode": "selfmanaged", 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_seq": 0, 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_epoch": 0, 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "pool_snaps": [], 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_snaps": "[]", 2020-03-09T00:25:06.094 INFO:teuthology.orchestra.run.smithi148.stdout: "quota_max_bytes": 0, 2020-03-09T00:25:06.095 INFO:teuthology.orchestra.run.smithi148.stdout: "quota_max_objects": 0, 2020-03-09T00:25:06.095 INFO:teuthology.orchestra.run.smithi148.stdout: "tiers": [], 2020-03-09T00:25:06.095 INFO:teuthology.orchestra.run.smithi148.stdout: "tier_of": -1, 2020-03-09T00:25:06.095 INFO:teuthology.orchestra.run.smithi148.stdout: "read_tier": -1, 2020-03-09T00:25:06.095 INFO:teuthology.orchestra.run.smithi148.stdout: "write_tier": -1, 2020-03-09T00:25:06.095 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_mode": "none", 2020-03-09T00:25:06.095 INFO:teuthology.orchestra.run.smithi148.stdout: "target_max_bytes": 0, 2020-03-09T00:25:06.096 INFO:teuthology.orchestra.run.smithi148.stdout: "target_max_objects": 0, 2020-03-09T00:25:06.096 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-09T00:25:06.096 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-09T00:25:06.096 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-09T00:25:06.096 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_min_flush_age": 0, 2020-03-09T00:25:06.096 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_min_evict_age": 0, 2020-03-09T00:25:06.096 INFO:teuthology.orchestra.run.smithi148.stdout: "erasure_code_profile": "", 2020-03-09T00:25:06.097 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_params": { 2020-03-09T00:25:06.097 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "none" 2020-03-09T00:25:06.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.097 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_period": 0, 2020-03-09T00:25:06.097 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_count": 0, 2020-03-09T00:25:06.097 INFO:teuthology.orchestra.run.smithi148.stdout: "use_gmt_hitset": true, 2020-03-09T00:25:06.097 INFO:teuthology.orchestra.run.smithi148.stdout: "min_read_recency_for_promote": 0, 2020-03-09T00:25:06.098 INFO:teuthology.orchestra.run.smithi148.stdout: "min_write_recency_for_promote": 0, 2020-03-09T00:25:06.098 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_grade_decay_rate": 0, 2020-03-09T00:25:06.098 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_search_last_n": 0, 2020-03-09T00:25:06.098 INFO:teuthology.orchestra.run.smithi148.stdout: "grade_table": [], 2020-03-09T00:25:06.098 INFO:teuthology.orchestra.run.smithi148.stdout: "stripe_width": 0, 2020-03-09T00:25:06.098 INFO:teuthology.orchestra.run.smithi148.stdout: "expected_num_objects": 0, 2020-03-09T00:25:06.098 INFO:teuthology.orchestra.run.smithi148.stdout: "fast_read": false, 2020-03-09T00:25:06.099 INFO:teuthology.orchestra.run.smithi148.stdout: "options": { 2020-03-09T00:25:06.099 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num_min": 1 2020-03-09T00:25:06.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.099 INFO:teuthology.orchestra.run.smithi148.stdout: "application_metadata": { 2020-03-09T00:25:06.099 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr_devicehealth": {} 2020-03-09T00:25:06.099 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.100 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 2, 2020-03-09T00:25:06.100 INFO:teuthology.orchestra.run.smithi148.stdout: "pool_name": "rbd", 2020-03-09T00:25:06.100 INFO:teuthology.orchestra.run.smithi148.stdout: "create_time": "2020-03-09T00:24:26.419136+0000", 2020-03-09T00:25:06.100 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": 1, 2020-03-09T00:25:06.100 INFO:teuthology.orchestra.run.smithi148.stdout: "flags_names": "hashpspool", 2020-03-09T00:25:06.100 INFO:teuthology.orchestra.run.smithi148.stdout: "type": 1, 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 2, 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "min_size": 1, 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_rule": 0, 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "object_hash": 2, 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_autoscale_mode": "off", 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num": 8, 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_placement_num": 8, 2020-03-09T00:25:06.101 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_placement_num_target": 8, 2020-03-09T00:25:06.102 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num_target": 8, 2020-03-09T00:25:06.102 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num_pending": 8, 2020-03-09T00:25:06.102 INFO:teuthology.orchestra.run.smithi148.stdout: "last_pg_merge_meta": { 2020-03-09T00:25:06.102 INFO:teuthology.orchestra.run.smithi148.stdout: "source_pgid": "0.0", 2020-03-09T00:25:06.102 INFO:teuthology.orchestra.run.smithi148.stdout: "ready_epoch": 0, 2020-03-09T00:25:06.102 INFO:teuthology.orchestra.run.smithi148.stdout: "last_epoch_started": 0, 2020-03-09T00:25:06.102 INFO:teuthology.orchestra.run.smithi148.stdout: "last_epoch_clean": 0, 2020-03-09T00:25:06.103 INFO:teuthology.orchestra.run.smithi148.stdout: "source_version": "0'0", 2020-03-09T00:25:06.103 INFO:teuthology.orchestra.run.smithi148.stdout: "target_version": "0'0" 2020-03-09T00:25:06.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.103 INFO:teuthology.orchestra.run.smithi148.stdout: "last_change": "29", 2020-03-09T00:25:06.103 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend": "0", 2020-03-09T00:25:06.103 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-09T00:25:06.103 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend_preluminous": "0", 2020-03-09T00:25:06.104 INFO:teuthology.orchestra.run.smithi148.stdout: "auid": 0, 2020-03-09T00:25:06.104 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_mode": "selfmanaged", 2020-03-09T00:25:06.104 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_seq": 0, 2020-03-09T00:25:06.104 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_epoch": 0, 2020-03-09T00:25:06.104 INFO:teuthology.orchestra.run.smithi148.stdout: "pool_snaps": [], 2020-03-09T00:25:06.104 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_snaps": "[]", 2020-03-09T00:25:06.104 INFO:teuthology.orchestra.run.smithi148.stdout: "quota_max_bytes": 0, 2020-03-09T00:25:06.105 INFO:teuthology.orchestra.run.smithi148.stdout: "quota_max_objects": 0, 2020-03-09T00:25:06.105 INFO:teuthology.orchestra.run.smithi148.stdout: "tiers": [], 2020-03-09T00:25:06.105 INFO:teuthology.orchestra.run.smithi148.stdout: "tier_of": -1, 2020-03-09T00:25:06.105 INFO:teuthology.orchestra.run.smithi148.stdout: "read_tier": -1, 2020-03-09T00:25:06.105 INFO:teuthology.orchestra.run.smithi148.stdout: "write_tier": -1, 2020-03-09T00:25:06.105 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_mode": "none", 2020-03-09T00:25:06.105 INFO:teuthology.orchestra.run.smithi148.stdout: "target_max_bytes": 0, 2020-03-09T00:25:06.106 INFO:teuthology.orchestra.run.smithi148.stdout: "target_max_objects": 0, 2020-03-09T00:25:06.106 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-09T00:25:06.106 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-09T00:25:06.106 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-09T00:25:06.106 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_min_flush_age": 0, 2020-03-09T00:25:06.106 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_min_evict_age": 0, 2020-03-09T00:25:06.107 INFO:teuthology.orchestra.run.smithi148.stdout: "erasure_code_profile": "", 2020-03-09T00:25:06.107 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_params": { 2020-03-09T00:25:06.107 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "none" 2020-03-09T00:25:06.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.107 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_period": 0, 2020-03-09T00:25:06.107 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_count": 0, 2020-03-09T00:25:06.108 INFO:teuthology.orchestra.run.smithi148.stdout: "use_gmt_hitset": true, 2020-03-09T00:25:06.108 INFO:teuthology.orchestra.run.smithi148.stdout: "min_read_recency_for_promote": 0, 2020-03-09T00:25:06.108 INFO:teuthology.orchestra.run.smithi148.stdout: "min_write_recency_for_promote": 0, 2020-03-09T00:25:06.108 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_grade_decay_rate": 0, 2020-03-09T00:25:06.108 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_search_last_n": 0, 2020-03-09T00:25:06.108 INFO:teuthology.orchestra.run.smithi148.stdout: "grade_table": [], 2020-03-09T00:25:06.109 INFO:teuthology.orchestra.run.smithi148.stdout: "stripe_width": 0, 2020-03-09T00:25:06.109 INFO:teuthology.orchestra.run.smithi148.stdout: "expected_num_objects": 0, 2020-03-09T00:25:06.109 INFO:teuthology.orchestra.run.smithi148.stdout: "fast_read": false, 2020-03-09T00:25:06.109 INFO:teuthology.orchestra.run.smithi148.stdout: "options": {}, 2020-03-09T00:25:06.109 INFO:teuthology.orchestra.run.smithi148.stdout: "application_metadata": { 2020-03-09T00:25:06.109 INFO:teuthology.orchestra.run.smithi148.stdout: "rbd": {} 2020-03-09T00:25:06.109 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.110 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 3, 2020-03-09T00:25:06.110 INFO:teuthology.orchestra.run.smithi148.stdout: "pool_name": "unique_pool_0", 2020-03-09T00:25:06.110 INFO:teuthology.orchestra.run.smithi148.stdout: "create_time": "2020-03-09T00:24:40.793412+0000", 2020-03-09T00:25:06.110 INFO:teuthology.orchestra.run.smithi148.stdout: "flags": 8193, 2020-03-09T00:25:06.111 INFO:teuthology.orchestra.run.smithi148.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2020-03-09T00:25:06.111 INFO:teuthology.orchestra.run.smithi148.stdout: "type": 3, 2020-03-09T00:25:06.111 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 7, 2020-03-09T00:25:06.111 INFO:teuthology.orchestra.run.smithi148.stdout: "min_size": 5, 2020-03-09T00:25:06.111 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_rule": 1, 2020-03-09T00:25:06.111 INFO:teuthology.orchestra.run.smithi148.stdout: "object_hash": 2, 2020-03-09T00:25:06.111 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_autoscale_mode": "off", 2020-03-09T00:25:06.112 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num": 16, 2020-03-09T00:25:06.112 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_placement_num": 16, 2020-03-09T00:25:06.112 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_placement_num_target": 16, 2020-03-09T00:25:06.112 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num_target": 16, 2020-03-09T00:25:06.112 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_num_pending": 16, 2020-03-09T00:25:06.112 INFO:teuthology.orchestra.run.smithi148.stdout: "last_pg_merge_meta": { 2020-03-09T00:25:06.112 INFO:teuthology.orchestra.run.smithi148.stdout: "source_pgid": "0.0", 2020-03-09T00:25:06.113 INFO:teuthology.orchestra.run.smithi148.stdout: "ready_epoch": 0, 2020-03-09T00:25:06.113 INFO:teuthology.orchestra.run.smithi148.stdout: "last_epoch_started": 0, 2020-03-09T00:25:06.113 INFO:teuthology.orchestra.run.smithi148.stdout: "last_epoch_clean": 0, 2020-03-09T00:25:06.113 INFO:teuthology.orchestra.run.smithi148.stdout: "source_version": "0'0", 2020-03-09T00:25:06.113 INFO:teuthology.orchestra.run.smithi148.stdout: "target_version": "0'0" 2020-03-09T00:25:06.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.114 INFO:teuthology.orchestra.run.smithi148.stdout: "last_change": "56", 2020-03-09T00:25:06.114 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend": "0", 2020-03-09T00:25:06.114 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-09T00:25:06.114 INFO:teuthology.orchestra.run.smithi148.stdout: "last_force_op_resend_preluminous": "0", 2020-03-09T00:25:06.114 INFO:teuthology.orchestra.run.smithi148.stdout: "auid": 0, 2020-03-09T00:25:06.114 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_mode": "selfmanaged", 2020-03-09T00:25:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_seq": 21, 2020-03-09T00:25:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: "snap_epoch": 56, 2020-03-09T00:25:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: "pool_snaps": [], 2020-03-09T00:25:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_snaps": "[]", 2020-03-09T00:25:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: "quota_max_bytes": 0, 2020-03-09T00:25:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: "quota_max_objects": 0, 2020-03-09T00:25:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: "tiers": [], 2020-03-09T00:25:06.116 INFO:teuthology.orchestra.run.smithi148.stdout: "tier_of": -1, 2020-03-09T00:25:06.116 INFO:teuthology.orchestra.run.smithi148.stdout: "read_tier": -1, 2020-03-09T00:25:06.116 INFO:teuthology.orchestra.run.smithi148.stdout: "write_tier": -1, 2020-03-09T00:25:06.116 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_mode": "none", 2020-03-09T00:25:06.117 INFO:teuthology.orchestra.run.smithi148.stdout: "target_max_bytes": 0, 2020-03-09T00:25:06.117 INFO:teuthology.orchestra.run.smithi148.stdout: "target_max_objects": 0, 2020-03-09T00:25:06.117 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-09T00:25:06.117 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-09T00:25:06.118 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-09T00:25:06.118 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_min_flush_age": 0, 2020-03-09T00:25:06.118 INFO:teuthology.orchestra.run.smithi148.stdout: "cache_min_evict_age": 0, 2020-03-09T00:25:06.118 INFO:teuthology.orchestra.run.smithi148.stdout: "erasure_code_profile": "shecprofile", 2020-03-09T00:25:06.118 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_params": { 2020-03-09T00:25:06.118 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "none" 2020-03-09T00:25:06.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.119 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_period": 0, 2020-03-09T00:25:06.119 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_count": 0, 2020-03-09T00:25:06.119 INFO:teuthology.orchestra.run.smithi148.stdout: "use_gmt_hitset": true, 2020-03-09T00:25:06.119 INFO:teuthology.orchestra.run.smithi148.stdout: "min_read_recency_for_promote": 0, 2020-03-09T00:25:06.119 INFO:teuthology.orchestra.run.smithi148.stdout: "min_write_recency_for_promote": 0, 2020-03-09T00:25:06.119 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_grade_decay_rate": 0, 2020-03-09T00:25:06.120 INFO:teuthology.orchestra.run.smithi148.stdout: "hit_set_search_last_n": 0, 2020-03-09T00:25:06.120 INFO:teuthology.orchestra.run.smithi148.stdout: "grade_table": [], 2020-03-09T00:25:06.120 INFO:teuthology.orchestra.run.smithi148.stdout: "stripe_width": 16384, 2020-03-09T00:25:06.120 INFO:teuthology.orchestra.run.smithi148.stdout: "expected_num_objects": 0, 2020-03-09T00:25:06.120 INFO:teuthology.orchestra.run.smithi148.stdout: "fast_read": false, 2020-03-09T00:25:06.120 INFO:teuthology.orchestra.run.smithi148.stdout: "options": {}, 2020-03-09T00:25:06.120 INFO:teuthology.orchestra.run.smithi148.stdout: "application_metadata": { 2020-03-09T00:25:06.121 INFO:teuthology.orchestra.run.smithi148.stdout: "rados": {} 2020-03-09T00:25:06.121 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.121 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.121 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2020-03-09T00:25:06.121 INFO:teuthology.orchestra.run.smithi148.stdout: "osds": [ 2020-03-09T00:25:06.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.122 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 0, 2020-03-09T00:25:06.122 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "d0be76d1-3ce6-4c23-9917-41b35482def3", 2020-03-09T00:25:06.122 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.122 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.122 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.122 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.122 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 54, 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.124 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.124 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6816", 2020-03-09T00:25:06.124 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.124 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.126 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6817", 2020-03-09T00:25:06.126 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.126 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.126 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.127 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.127 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.127 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.127 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6818", 2020-03-09T00:25:06.127 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6819", 2020-03-09T00:25:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.128 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.129 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.129 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.129 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.129 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6822", 2020-03-09T00:25:06.130 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.130 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.130 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6824", 2020-03-09T00:25:06.130 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.130 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.131 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.131 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.131 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.131 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.131 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6820", 2020-03-09T00:25:06.132 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.132 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.132 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6821", 2020-03-09T00:25:06.132 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27385 2020-03-09T00:25:06.133 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.133 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.133 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6817/27385", 2020-03-09T00:25:06.133 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.148:6819/27385", 2020-03-09T00:25:06.133 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.148:6824/27385", 2020-03-09T00:25:06.134 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.148:6821/27385", 2020-03-09T00:25:06.134 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.134 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.135 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.135 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.135 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 1, 2020-03-09T00:25:06.136 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "cee70020-657e-4b62-89cb-226f760442f5", 2020-03-09T00:25:06.136 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.136 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.136 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.136 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.136 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.136 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.137 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.137 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 0, 2020-03-09T00:25:06.137 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.137 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.137 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.137 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6808", 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6809", 2020-03-09T00:25:06.138 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.139 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.139 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.139 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.139 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.140 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.140 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6810", 2020-03-09T00:25:06.140 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.140 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.140 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6811", 2020-03-09T00:25:06.141 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.141 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.141 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.141 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.141 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.142 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.142 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6814", 2020-03-09T00:25:06.142 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.142 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.143 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6815", 2020-03-09T00:25:06.143 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.143 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.143 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.143 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.143 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.144 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.144 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6812", 2020-03-09T00:25:06.144 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.144 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.145 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6813", 2020-03-09T00:25:06.145 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27629 2020-03-09T00:25:06.145 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.145 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.145 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.50:6809/27629", 2020-03-09T00:25:06.145 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.50:6811/27629", 2020-03-09T00:25:06.146 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.50:6815/27629", 2020-03-09T00:25:06.146 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.50:6813/27629", 2020-03-09T00:25:06.146 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.146 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.146 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.146 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.147 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 2, 2020-03-09T00:25:06.147 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "4103b4d7-7a38-48e4-8403-3549f5b3fda2", 2020-03-09T00:25:06.147 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.147 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.147 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.148 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.148 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.148 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.148 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.148 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 0, 2020-03-09T00:25:06.148 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.148 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.149 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.149 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.150 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.150 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6800", 2020-03-09T00:25:06.150 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.151 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.151 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6801", 2020-03-09T00:25:06.152 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.152 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.152 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.152 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.153 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.153 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.153 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6802", 2020-03-09T00:25:06.154 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.154 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.154 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6803", 2020-03-09T00:25:06.155 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.155 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.155 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.156 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.156 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.156 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.157 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6806", 2020-03-09T00:25:06.157 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.157 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.158 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6807", 2020-03-09T00:25:06.158 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.158 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.158 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.159 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.159 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.160 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.160 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6804", 2020-03-09T00:25:06.160 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.161 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.161 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6805", 2020-03-09T00:25:06.161 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26980 2020-03-09T00:25:06.162 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.162 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.162 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.139:6801/26980", 2020-03-09T00:25:06.163 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.139:6803/26980", 2020-03-09T00:25:06.163 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.139:6807/26980", 2020-03-09T00:25:06.163 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.139:6805/26980", 2020-03-09T00:25:06.163 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.164 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.164 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.164 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 3, 2020-03-09T00:25:06.165 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "ce80df27-1b2e-41e1-9ded-a012ad488591", 2020-03-09T00:25:06.165 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.165 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.166 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.166 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.166 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.166 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.167 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.167 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 32, 2020-03-09T00:25:06.167 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.167 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.168 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.168 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.168 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.169 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6802", 2020-03-09T00:25:06.169 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.170 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.170 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6803", 2020-03-09T00:25:06.170 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.171 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.171 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.171 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.172 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.172 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.172 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6804", 2020-03-09T00:25:06.173 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.173 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.174 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6805", 2020-03-09T00:25:06.174 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.174 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.174 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.175 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.175 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.176 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.176 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6808", 2020-03-09T00:25:06.176 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.177 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.177 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6809", 2020-03-09T00:25:06.177 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.177 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.178 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.178 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.178 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.179 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.179 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6806", 2020-03-09T00:25:06.179 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.180 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.180 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6807", 2020-03-09T00:25:06.181 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28012 2020-03-09T00:25:06.181 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.181 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.181 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.104:6803/28012", 2020-03-09T00:25:06.181 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.104:6805/28012", 2020-03-09T00:25:06.182 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.104:6809/28012", 2020-03-09T00:25:06.182 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.104:6807/28012", 2020-03-09T00:25:06.182 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.182 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.182 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.183 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.183 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 4, 2020-03-09T00:25:06.184 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "8ff5a871-6f10-4e55-ac60-33bdc11c0292", 2020-03-09T00:25:06.184 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.184 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.184 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.184 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.184 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.185 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.185 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.185 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 54, 2020-03-09T00:25:06.185 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.185 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.186 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.186 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.186 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.186 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6800", 2020-03-09T00:25:06.186 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.187 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.187 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6801", 2020-03-09T00:25:06.187 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.187 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.188 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.188 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.188 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.188 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.188 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6802", 2020-03-09T00:25:06.189 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.189 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.189 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6803", 2020-03-09T00:25:06.189 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.189 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.190 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.190 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.190 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.190 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.191 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6806", 2020-03-09T00:25:06.191 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.191 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.192 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6807", 2020-03-09T00:25:06.192 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.192 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.192 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.192 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.192 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.193 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.193 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6804", 2020-03-09T00:25:06.193 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.193 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6805", 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27365 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6801/27365", 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.148:6803/27365", 2020-03-09T00:25:06.194 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.148:6807/27365", 2020-03-09T00:25:06.195 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.148:6805/27365", 2020-03-09T00:25:06.195 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.195 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.195 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.195 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.196 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 5, 2020-03-09T00:25:06.196 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "5122cc94-7217-4d84-8b81-11383b48c416", 2020-03-09T00:25:06.196 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.196 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 0, 2020-03-09T00:25:06.196 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2020-03-09T00:25:06.196 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.196 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.197 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.197 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.197 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 45, 2020-03-09T00:25:06.197 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.197 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.197 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.197 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6800", 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.198 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6801", 2020-03-09T00:25:06.199 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.199 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.199 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.199 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.199 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.200 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.200 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6802", 2020-03-09T00:25:06.200 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.200 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.200 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6803", 2020-03-09T00:25:06.201 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.201 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.201 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.201 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.201 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6806", 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.202 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6807", 2020-03-09T00:25:06.203 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.203 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.203 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.203 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.203 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.204 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.204 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6804", 2020-03-09T00:25:06.204 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.204 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.204 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6805", 2020-03-09T00:25:06.205 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27630 2020-03-09T00:25:06.205 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.205 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.205 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.50:6801/27630", 2020-03-09T00:25:06.205 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.50:6803/27630", 2020-03-09T00:25:06.205 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.50:6807/27630", 2020-03-09T00:25:06.206 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.50:6805/27630", 2020-03-09T00:25:06.206 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.206 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.206 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.206 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.206 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.207 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 6, 2020-03-09T00:25:06.207 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "1bea4f99-5e2a-4fbc-bac3-8cb45d678742", 2020-03-09T00:25:06.207 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.207 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.207 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.207 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.208 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.208 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.208 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.208 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 54, 2020-03-09T00:25:06.208 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.208 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.209 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.209 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.209 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.209 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.209 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6809", 2020-03-09T00:25:06.209 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.209 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.210 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.210 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.210 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6811", 2020-03-09T00:25:06.210 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.210 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.210 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.211 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.211 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.211 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.211 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.211 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.211 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6813", 2020-03-09T00:25:06.212 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.212 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.212 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.212 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.212 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6815", 2020-03-09T00:25:06.212 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.212 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6823", 2020-03-09T00:25:06.214 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.214 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.214 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.214 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.214 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6826", 2020-03-09T00:25:06.214 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.214 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.215 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.215 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.215 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.215 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.216 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.216 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6817", 2020-03-09T00:25:06.216 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.216 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.216 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.216 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.216 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6820", 2020-03-09T00:25:06.217 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26979 2020-03-09T00:25:06.217 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.217 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.217 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.217 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.139:6811/26979", 2020-03-09T00:25:06.217 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.139:6815/26979", 2020-03-09T00:25:06.217 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.139:6826/26979", 2020-03-09T00:25:06.218 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.139:6820/26979", 2020-03-09T00:25:06.218 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.218 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.218 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.218 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.218 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.218 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.219 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 7, 2020-03-09T00:25:06.219 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "f6a853dd-1462-4f8c-83d6-701258d78519", 2020-03-09T00:25:06.219 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.219 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.219 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.219 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.219 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.220 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.220 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.220 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 47, 2020-03-09T00:25:06.220 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.220 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.221 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.221 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.221 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.221 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.221 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6818", 2020-03-09T00:25:06.222 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.222 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.222 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.222 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.222 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6819", 2020-03-09T00:25:06.222 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6820", 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6822", 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.224 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.225 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.225 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.225 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.225 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.225 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.225 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.225 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6827", 2020-03-09T00:25:06.226 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.226 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.226 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.226 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.226 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6829", 2020-03-09T00:25:06.226 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.226 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.227 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.227 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.227 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.227 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.227 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.227 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.228 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6824", 2020-03-09T00:25:06.228 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.228 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.228 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.228 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.228 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6826", 2020-03-09T00:25:06.228 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28042 2020-03-09T00:25:06.229 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.229 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.229 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.229 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.104:6819/28042", 2020-03-09T00:25:06.229 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.104:6822/28042", 2020-03-09T00:25:06.229 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.104:6829/28042", 2020-03-09T00:25:06.230 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.104:6826/28042", 2020-03-09T00:25:06.230 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.230 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.230 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.230 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.230 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.231 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.231 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 8, 2020-03-09T00:25:06.231 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "b9911260-c1d0-4563-892b-35a9e1c47a9b", 2020-03-09T00:25:06.231 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.231 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.232 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.232 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.232 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.232 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.232 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.232 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 32, 2020-03-09T00:25:06.233 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.233 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.233 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.233 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.233 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.234 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6808", 2020-03-09T00:25:06.234 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.234 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.234 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.234 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.234 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6809", 2020-03-09T00:25:06.234 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.235 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.235 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.235 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.235 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.236 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.236 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6810", 2020-03-09T00:25:06.236 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6811", 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.237 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.238 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.238 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6814", 2020-03-09T00:25:06.238 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.238 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.239 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6815", 2020-03-09T00:25:06.239 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.239 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.239 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.239 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.239 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6812", 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.240 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6813", 2020-03-09T00:25:06.241 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27364 2020-03-09T00:25:06.241 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.241 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.241 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6809/27364", 2020-03-09T00:25:06.242 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.148:6811/27364", 2020-03-09T00:25:06.242 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.148:6815/27364", 2020-03-09T00:25:06.242 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.148:6813/27364", 2020-03-09T00:25:06.242 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.242 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.242 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.243 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.243 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.243 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.243 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 9, 2020-03-09T00:25:06.243 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "d7ebf4f6-a2b8-4076-83dd-504172481e32", 2020-03-09T00:25:06.243 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.243 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.244 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.244 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.244 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.244 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.244 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.244 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 48, 2020-03-09T00:25:06.245 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.245 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.245 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.245 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.245 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.245 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6821", 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6823", 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.246 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.247 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.247 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.247 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.247 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.247 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.247 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.247 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6825", 2020-03-09T00:25:06.248 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.248 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.248 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.248 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.248 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6827", 2020-03-09T00:25:06.248 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.249 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.249 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.249 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.249 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.249 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.249 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.250 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.250 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6830", 2020-03-09T00:25:06.250 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.250 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.250 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.250 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.251 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6831", 2020-03-09T00:25:06.251 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.251 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.252 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.252 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.253 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.253 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.253 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.253 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.253 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6828", 2020-03-09T00:25:06.254 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.254 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.254 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.254 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.254 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6829", 2020-03-09T00:25:06.254 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27643 2020-03-09T00:25:06.254 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.50:6823/27643", 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.50:6827/27643", 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.50:6831/27643", 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.50:6829/27643", 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.255 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.256 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.256 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.256 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.256 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.256 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 10, 2020-03-09T00:25:06.256 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "cb76df3e-6ea9-4ee6-8b03-84e3cd329c69", 2020-03-09T00:25:06.256 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.257 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.257 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.257 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.257 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.257 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.257 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 23, 2020-03-09T00:25:06.257 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 53, 2020-03-09T00:25:06.258 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.258 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.258 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.258 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.258 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.258 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.259 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6808", 2020-03-09T00:25:06.259 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.259 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.259 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.259 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.259 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6810", 2020-03-09T00:25:06.259 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.260 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.260 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.260 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.260 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.260 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.260 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.260 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.261 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6812", 2020-03-09T00:25:06.261 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.261 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.261 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.261 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.261 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6814", 2020-03-09T00:25:06.261 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.262 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.266 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.266 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.267 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.267 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.267 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6822", 2020-03-09T00:25:06.268 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.268 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.268 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.268 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.268 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6825", 2020-03-09T00:25:06.268 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.268 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.269 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6816", 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6819", 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26978 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.270 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.271 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.139:6810/26978", 2020-03-09T00:25:06.271 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.139:6814/26978", 2020-03-09T00:25:06.271 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.139:6825/26978", 2020-03-09T00:25:06.271 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.139:6819/26978", 2020-03-09T00:25:06.271 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.271 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.272 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.272 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.272 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.272 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.272 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 11, 2020-03-09T00:25:06.273 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "8bda27ba-cf17-472c-9a21-02dcaefab68f", 2020-03-09T00:25:06.273 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.273 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.273 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.273 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.273 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.273 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.274 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.274 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 48, 2020-03-09T00:25:06.274 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.274 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.274 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.274 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.274 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.275 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.275 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6810", 2020-03-09T00:25:06.275 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.275 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.275 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.275 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.275 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6811", 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.276 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.277 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6812", 2020-03-09T00:25:06.277 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.277 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.277 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.277 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.277 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6813", 2020-03-09T00:25:06.277 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.278 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.278 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.278 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.278 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.278 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.278 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.278 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6816", 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6817", 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.279 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.280 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.280 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.280 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.280 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6814", 2020-03-09T00:25:06.281 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.281 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.281 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.281 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.281 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6815", 2020-03-09T00:25:06.281 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28013 2020-03-09T00:25:06.281 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.282 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.282 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.104:6811/28013", 2020-03-09T00:25:06.282 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.104:6813/28013", 2020-03-09T00:25:06.282 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.104:6817/28013", 2020-03-09T00:25:06.282 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.104:6815/28013", 2020-03-09T00:25:06.282 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 12, 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "d861da04-0be8-434b-9ed8-c1ec97fca7bd", 2020-03-09T00:25:06.283 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.284 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 54, 2020-03-09T00:25:06.285 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.285 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.285 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.285 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.285 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.285 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.285 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6823", 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6825", 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.286 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.287 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.287 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.287 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.287 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.287 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.287 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6826", 2020-03-09T00:25:06.287 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6827", 2020-03-09T00:25:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.289 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.289 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.289 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.289 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.289 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6830", 2020-03-09T00:25:06.289 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.290 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.290 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.290 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6831", 2020-03-09T00:25:06.290 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.290 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.290 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.291 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.291 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.291 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.291 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.291 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.291 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6828", 2020-03-09T00:25:06.291 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.292 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.292 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.292 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.292 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6829", 2020-03-09T00:25:06.292 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27395 2020-03-09T00:25:06.292 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.293 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.293 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.293 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6825/27395", 2020-03-09T00:25:06.293 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.148:6827/27395", 2020-03-09T00:25:06.293 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.148:6831/27395", 2020-03-09T00:25:06.293 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.148:6829/27395", 2020-03-09T00:25:06.293 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 13, 2020-03-09T00:25:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "4ba3410e-df5b-4529-9ebe-34c15f68bc9d", 2020-03-09T00:25:06.295 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.295 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 0, 2020-03-09T00:25:06.295 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2020-03-09T00:25:06.295 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.295 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.295 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.295 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.296 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 32, 2020-03-09T00:25:06.296 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.296 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.296 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.296 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.296 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.297 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.297 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6816", 2020-03-09T00:25:06.298 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.298 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.298 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.299 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.299 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6817", 2020-03-09T00:25:06.299 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.300 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.300 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6818", 2020-03-09T00:25:06.301 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6819", 2020-03-09T00:25:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.302 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6824", 2020-03-09T00:25:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.304 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.304 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.304 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6826", 2020-03-09T00:25:06.304 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.304 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.305 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.305 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.305 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.305 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6820", 2020-03-09T00:25:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.306 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.307 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.50:6822", 2020-03-09T00:25:06.307 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 27631 2020-03-09T00:25:06.307 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.307 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.307 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.307 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.50:6817/27631", 2020-03-09T00:25:06.308 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.50:6819/27631", 2020-03-09T00:25:06.308 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.50:6826/27631", 2020-03-09T00:25:06.308 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.50:6822/27631", 2020-03-09T00:25:06.308 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.308 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.308 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.308 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.309 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.309 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.309 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 14, 2020-03-09T00:25:06.309 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "1ba69402-6d5c-491e-ac86-747517533a8b", 2020-03-09T00:25:06.309 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.309 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.310 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.310 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.310 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.310 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.311 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.311 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 54, 2020-03-09T00:25:06.311 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.311 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.311 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6818", 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.312 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.313 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6821", 2020-03-09T00:25:06.313 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.313 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.313 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.313 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.313 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.314 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.314 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.314 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.314 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6824", 2020-03-09T00:25:06.314 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.314 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.315 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.315 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.315 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6827", 2020-03-09T00:25:06.315 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.315 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.315 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.318 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.318 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.319 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.319 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.319 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.319 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6830", 2020-03-09T00:25:06.319 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.320 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.320 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.320 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.320 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6831", 2020-03-09T00:25:06.320 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.320 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.321 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.321 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.321 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.321 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.321 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.321 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.322 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6828", 2020-03-09T00:25:06.322 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.322 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.322 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.322 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.322 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.139:6829", 2020-03-09T00:25:06.323 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 26981 2020-03-09T00:25:06.323 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.323 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.323 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.323 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.139:6821/26981", 2020-03-09T00:25:06.324 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.139:6827/26981", 2020-03-09T00:25:06.324 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.139:6831/26981", 2020-03-09T00:25:06.324 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.139:6829/26981", 2020-03-09T00:25:06.324 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.324 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.324 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.324 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.325 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.325 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.325 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 15, 2020-03-09T00:25:06.325 INFO:teuthology.orchestra.run.smithi148.stdout: "uuid": "a3566380-0c4a-403f-8533-b2d0dcfa8506", 2020-03-09T00:25:06.325 INFO:teuthology.orchestra.run.smithi148.stdout: "up": 1, 2020-03-09T00:25:06.325 INFO:teuthology.orchestra.run.smithi148.stdout: "in": 1, 2020-03-09T00:25:06.325 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 1, 2020-03-09T00:25:06.326 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_affinity": 1, 2020-03-09T00:25:06.326 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_begin": 0, 2020-03-09T00:25:06.326 INFO:teuthology.orchestra.run.smithi148.stdout: "last_clean_end": 0, 2020-03-09T00:25:06.326 INFO:teuthology.orchestra.run.smithi148.stdout: "up_from": 24, 2020-03-09T00:25:06.326 INFO:teuthology.orchestra.run.smithi148.stdout: "up_thru": 51, 2020-03-09T00:25:06.326 INFO:teuthology.orchestra.run.smithi148.stdout: "down_at": 0, 2020-03-09T00:25:06.326 INFO:teuthology.orchestra.run.smithi148.stdout: "lost_at": 0, 2020-03-09T00:25:06.327 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2020-03-09T00:25:06.327 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.327 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.327 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.327 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6821", 2020-03-09T00:25:06.327 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.327 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.328 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.328 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.328 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6823", 2020-03-09T00:25:06.328 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.328 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.328 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.329 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.329 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addrs": { 2020-03-09T00:25:06.329 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.329 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.330 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.330 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6825", 2020-03-09T00:25:06.330 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.330 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.330 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.331 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.331 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6828", 2020-03-09T00:25:06.331 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.331 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.332 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.332 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.332 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addrs": { 2020-03-09T00:25:06.332 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.332 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.332 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.333 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6832", 2020-03-09T00:25:06.333 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.333 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.333 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.333 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.333 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6833", 2020-03-09T00:25:06.333 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.334 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.334 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.334 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.334 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addrs": { 2020-03-09T00:25:06.334 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2020-03-09T00:25:06.335 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.335 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v2", 2020-03-09T00:25:06.335 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6830", 2020-03-09T00:25:06.335 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.335 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.335 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.336 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2020-03-09T00:25:06.336 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.104:6831", 2020-03-09T00:25:06.336 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 28043 2020-03-09T00:25:06.336 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.336 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.337 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.337 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.104:6823/28043", 2020-03-09T00:25:06.337 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_addr": "172.21.15.104:6828/28043", 2020-03-09T00:25:06.337 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_back_addr": "172.21.15.104:6833/28043", 2020-03-09T00:25:06.337 INFO:teuthology.orchestra.run.smithi148.stdout: "heartbeat_front_addr": "172.21.15.104:6831/28043", 2020-03-09T00:25:06.337 INFO:teuthology.orchestra.run.smithi148.stdout: "state": [ 2020-03-09T00:25:06.338 INFO:teuthology.orchestra.run.smithi148.stdout: "exists", 2020-03-09T00:25:06.338 INFO:teuthology.orchestra.run.smithi148.stdout: "up" 2020-03-09T00:25:06.338 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.338 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.338 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2020-03-09T00:25:06.338 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_xinfo": [ 2020-03-09T00:25:06.338 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.339 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 0, 2020-03-09T00:25:06.339 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.339 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.339 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.340 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.340 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.341 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.085664+0000", 2020-03-09T00:25:06.341 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.341 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.341 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.341 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 1, 2020-03-09T00:25:06.342 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.342 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.342 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.342 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.342 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.342 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.111969+0000", 2020-03-09T00:25:06.342 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 2, 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.343 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.344 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.060238+0000", 2020-03-09T00:25:06.344 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.344 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.344 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.344 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 3, 2020-03-09T00:25:06.344 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.344 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.345 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.345 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.345 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.345 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:18.999353+0000", 2020-03-09T00:25:06.345 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.345 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.345 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 4, 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.026007+0000", 2020-03-09T00:25:06.346 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.347 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.347 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.347 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 5, 2020-03-09T00:25:06.347 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.347 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.347 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.347 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 65536, 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.046589+0000", 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 6, 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.348 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.085042+0000", 2020-03-09T00:25:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.349 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 7, 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.111926+0000", 2020-03-09T00:25:06.350 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 8, 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.351 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.352 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.072146+0000", 2020-03-09T00:25:06.352 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.352 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.352 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.352 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 9, 2020-03-09T00:25:06.352 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.352 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.353 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.353 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.353 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.353 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.062486+0000", 2020-03-09T00:25:06.353 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.353 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 10, 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:18.773795+0000", 2020-03-09T00:25:06.355 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.355 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.355 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.355 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 11, 2020-03-09T00:25:06.355 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.355 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.355 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.033423+0000", 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 12, 2020-03-09T00:25:06.356 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.138144+0000", 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.357 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 13, 2020-03-09T00:25:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 65536, 2020-03-09T00:25:06.358 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.118617+0000", 2020-03-09T00:25:06.359 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.359 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.359 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.359 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 14, 2020-03-09T00:25:06.359 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.360 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.360 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.360 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.360 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.360 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.039130+0000", 2020-03-09T00:25:06.360 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.360 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": 15, 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: "down_stamp": "0.000000", 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_probability": 0, 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: "laggy_interval": 0, 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: "features": 4540138292836696063, 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: "old_weight": 0, 2020-03-09T00:25:06.361 INFO:teuthology.orchestra.run.smithi148.stdout: "last_purged_snaps_scrub": "2020-03-09T00:24:19.098144+0000", 2020-03-09T00:25:06.362 INFO:teuthology.orchestra.run.smithi148.stdout: "dead_epoch": 0 2020-03-09T00:25:06.362 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.362 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2020-03-09T00:25:06.362 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_upmap": [], 2020-03-09T00:25:06.362 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_upmap_items": [], 2020-03-09T00:25:06.362 INFO:teuthology.orchestra.run.smithi148.stdout: "pg_temp": [], 2020-03-09T00:25:06.362 INFO:teuthology.orchestra.run.smithi148.stdout: "primary_temp": [], 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "blacklist": {}, 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "erasure_code_profiles": { 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "default": { 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "crush-failure-domain": "osd", 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "k": "2", 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "m": "1", 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "plugin": "jerasure", 2020-03-09T00:25:06.363 INFO:teuthology.orchestra.run.smithi148.stdout: "ruleset-failure-domain": "osd", 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: "technique": "reed_sol_van" 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: "shecprofile": { 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: "c": "2", 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: "crush-device-class": "", 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: "crush-failure-domain": "osd", 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: "crush-root": "default", 2020-03-09T00:25:06.364 INFO:teuthology.orchestra.run.smithi148.stdout: "k": "4", 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: "m": "3", 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "shecprofile", 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: "plugin": "shec", 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: "technique": "multiple" 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_snaps_queue": [ 2020-03-09T00:25:06.365 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.366 INFO:teuthology.orchestra.run.smithi148.stdout: "pool": 3, 2020-03-09T00:25:06.366 INFO:teuthology.orchestra.run.smithi148.stdout: "snaps": [ 2020-03-09T00:25:06.366 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.366 INFO:teuthology.orchestra.run.smithi148.stdout: "begin": 4, 2020-03-09T00:25:06.366 INFO:teuthology.orchestra.run.smithi148.stdout: "length": 1 2020-03-09T00:25:06.366 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.366 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: "begin": 9, 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: "length": 1 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: "begin": 11, 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: "length": 1 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.367 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:06.368 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:06.368 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2020-03-09T00:25:06.368 INFO:teuthology.orchestra.run.smithi148.stdout: "new_removed_snaps": [], 2020-03-09T00:25:06.368 INFO:teuthology.orchestra.run.smithi148.stdout: "new_purged_snaps": [], 2020-03-09T00:25:06.368 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_node_flags": {}, 2020-03-09T00:25:06.368 INFO:teuthology.orchestra.run.smithi148.stdout: "device_class_flags": {} 2020-03-09T00:25:06.368 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:06.370 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:06.370 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:06.372 INFO:tasks.thrashosds.thrasher:j is {u'nearfull_ratio': 0.8500000238418579, u'backfillfull_ratio': 0.8999999761581421, u'last_in_change': u'2020-03-09T00:25:00.770574+0000', u'full_ratio': 0.949999988079071, u'cluster_snapshot': u'', u'pg_upmap': [], u'new_purged_snaps': [], u'erasure_code_profiles': {u'default': {u'plugin': u'jerasure', u'crush-failure-domain': u'osd', u'k': u'2', u'technique': u'reed_sol_van', u'm': u'1', u'ruleset-failure-domain': u'osd'}, u'shecprofile': {u'c': u'2', u'name': u'shecprofile', u'plugin': u'shec', u'crush-root': u'default', u'k': u'4', u'technique': u'multiple', u'm': u'3', u'crush-failure-domain': u'osd', u'crush-device-class': u''}}, u'osds': [{u'heartbeat_back_addr': u'172.21.15.148:6824/27385', u'uuid': u'd0be76d1-3ce6-4c23-9917-41b35482def3', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.148:6819/27385', u'heartbeat_front_addr': u'172.21.15.148:6821/27385', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27385, u'type': u'v2', u'addr': u'172.21.15.148:6816'}, {u'nonce': 27385, u'type': u'v1', u'addr': u'172.21.15.148:6817'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27385, u'type': u'v2', u'addr': u'172.21.15.148:6818'}, {u'nonce': 27385, u'type': u'v1', u'addr': u'172.21.15.148:6819'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.148:6817/27385', u'up_thru': 54, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27385, u'type': u'v2', u'addr': u'172.21.15.148:6822'}, {u'nonce': 27385, u'type': u'v1', u'addr': u'172.21.15.148:6824'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27385, u'type': u'v2', u'addr': u'172.21.15.148:6820'}, {u'nonce': 27385, u'type': u'v1', u'addr': u'172.21.15.148:6821'}]}, u'osd': 0}, {u'heartbeat_back_addr': u'172.21.15.50:6815/27629', u'uuid': u'cee70020-657e-4b62-89cb-226f760442f5', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.50:6811/27629', u'heartbeat_front_addr': u'172.21.15.50:6813/27629', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27629, u'type': u'v2', u'addr': u'172.21.15.50:6808'}, {u'nonce': 27629, u'type': u'v1', u'addr': u'172.21.15.50:6809'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27629, u'type': u'v2', u'addr': u'172.21.15.50:6810'}, {u'nonce': 27629, u'type': u'v1', u'addr': u'172.21.15.50:6811'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.50:6809/27629', u'up_thru': 0, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27629, u'type': u'v2', u'addr': u'172.21.15.50:6814'}, {u'nonce': 27629, u'type': u'v1', u'addr': u'172.21.15.50:6815'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27629, u'type': u'v2', u'addr': u'172.21.15.50:6812'}, {u'nonce': 27629, u'type': u'v1', u'addr': u'172.21.15.50:6813'}]}, u'osd': 1}, {u'heartbeat_back_addr': u'172.21.15.139:6807/26980', u'uuid': u'4103b4d7-7a38-48e4-8403-3549f5b3fda2', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.139:6803/26980', u'heartbeat_front_addr': u'172.21.15.139:6805/26980', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 26980, u'type': u'v2', u'addr': u'172.21.15.139:6800'}, {u'nonce': 26980, u'type': u'v1', u'addr': u'172.21.15.139:6801'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 26980, u'type': u'v2', u'addr': u'172.21.15.139:6802'}, {u'nonce': 26980, u'type': u'v1', u'addr': u'172.21.15.139:6803'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.139:6801/26980', u'up_thru': 0, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 26980, u'type': u'v2', u'addr': u'172.21.15.139:6806'}, {u'nonce': 26980, u'type': u'v1', u'addr': u'172.21.15.139:6807'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 26980, u'type': u'v2', u'addr': u'172.21.15.139:6804'}, {u'nonce': 26980, u'type': u'v1', u'addr': u'172.21.15.139:6805'}]}, u'osd': 2}, {u'heartbeat_back_addr': u'172.21.15.104:6809/28012', u'uuid': u'ce80df27-1b2e-41e1-9ded-a012ad488591', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.104:6805/28012', u'heartbeat_front_addr': u'172.21.15.104:6807/28012', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 28012, u'type': u'v2', u'addr': u'172.21.15.104:6802'}, {u'nonce': 28012, u'type': u'v1', u'addr': u'172.21.15.104:6803'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 28012, u'type': u'v2', u'addr': u'172.21.15.104:6804'}, {u'nonce': 28012, u'type': u'v1', u'addr': u'172.21.15.104:6805'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.104:6803/28012', u'up_thru': 32, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 28012, u'type': u'v2', u'addr': u'172.21.15.104:6808'}, {u'nonce': 28012, u'type': u'v1', u'addr': u'172.21.15.104:6809'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 28012, u'type': u'v2', u'addr': u'172.21.15.104:6806'}, {u'nonce': 28012, u'type': u'v1', u'addr': u'172.21.15.104:6807'}]}, u'osd': 3}, {u'heartbeat_back_addr': u'172.21.15.148:6807/27365', u'uuid': u'8ff5a871-6f10-4e55-ac60-33bdc11c0292', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.148:6803/27365', u'heartbeat_front_addr': u'172.21.15.148:6805/27365', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27365, u'type': u'v2', u'addr': u'172.21.15.148:6800'}, {u'nonce': 27365, u'type': u'v1', u'addr': u'172.21.15.148:6801'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27365, u'type': u'v2', u'addr': u'172.21.15.148:6802'}, {u'nonce': 27365, u'type': u'v1', u'addr': u'172.21.15.148:6803'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.148:6801/27365', u'up_thru': 54, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27365, u'type': u'v2', u'addr': u'172.21.15.148:6806'}, {u'nonce': 27365, u'type': u'v1', u'addr': u'172.21.15.148:6807'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27365, u'type': u'v2', u'addr': u'172.21.15.148:6804'}, {u'nonce': 27365, u'type': u'v1', u'addr': u'172.21.15.148:6805'}]}, u'osd': 4}, {u'heartbeat_back_addr': u'172.21.15.50:6807/27630', u'uuid': u'5122cc94-7217-4d84-8b81-11383b48c416', u'weight': 0, u'up_from': 24, u'cluster_addr': u'172.21.15.50:6803/27630', u'heartbeat_front_addr': u'172.21.15.50:6805/27630', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27630, u'type': u'v2', u'addr': u'172.21.15.50:6800'}, {u'nonce': 27630, u'type': u'v1', u'addr': u'172.21.15.50:6801'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27630, u'type': u'v2', u'addr': u'172.21.15.50:6802'}, {u'nonce': 27630, u'type': u'v1', u'addr': u'172.21.15.50:6803'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 0, u'public_addr': u'172.21.15.50:6801/27630', u'up_thru': 45, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27630, u'type': u'v2', u'addr': u'172.21.15.50:6806'}, {u'nonce': 27630, u'type': u'v1', u'addr': u'172.21.15.50:6807'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27630, u'type': u'v2', u'addr': u'172.21.15.50:6804'}, {u'nonce': 27630, u'type': u'v1', u'addr': u'172.21.15.50:6805'}]}, u'osd': 5}, {u'heartbeat_back_addr': u'172.21.15.139:6826/26979', u'uuid': u'1bea4f99-5e2a-4fbc-bac3-8cb45d678742', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.139:6815/26979', u'heartbeat_front_addr': u'172.21.15.139:6820/26979', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 26979, u'type': u'v2', u'addr': u'172.21.15.139:6809'}, {u'nonce': 26979, u'type': u'v1', u'addr': u'172.21.15.139:6811'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 26979, u'type': u'v2', u'addr': u'172.21.15.139:6813'}, {u'nonce': 26979, u'type': u'v1', u'addr': u'172.21.15.139:6815'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.139:6811/26979', u'up_thru': 54, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 26979, u'type': u'v2', u'addr': u'172.21.15.139:6823'}, {u'nonce': 26979, u'type': u'v1', u'addr': u'172.21.15.139:6826'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 26979, u'type': u'v2', u'addr': u'172.21.15.139:6817'}, {u'nonce': 26979, u'type': u'v1', u'addr': u'172.21.15.139:6820'}]}, u'osd': 6}, {u'heartbeat_back_addr': u'172.21.15.104:6829/28042', u'uuid': u'f6a853dd-1462-4f8c-83d6-701258d78519', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.104:6822/28042', u'heartbeat_front_addr': u'172.21.15.104:6826/28042', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 28042, u'type': u'v2', u'addr': u'172.21.15.104:6818'}, {u'nonce': 28042, u'type': u'v1', u'addr': u'172.21.15.104:6819'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 28042, u'type': u'v2', u'addr': u'172.21.15.104:6820'}, {u'nonce': 28042, u'type': u'v1', u'addr': u'172.21.15.104:6822'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.104:6819/28042', u'up_thru': 47, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 28042, u'type': u'v2', u'addr': u'172.21.15.104:6827'}, {u'nonce': 28042, u'type': u'v1', u'addr': u'172.21.15.104:6829'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 28042, u'type': u'v2', u'addr': u'172.21.15.104:6824'}, {u'nonce': 28042, u'type': u'v1', u'addr': u'172.21.15.104:6826'}]}, u'osd': 7}, {u'heartbeat_back_addr': u'172.21.15.148:6815/27364', u'uuid': u'b9911260-c1d0-4563-892b-35a9e1c47a9b', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.148:6811/27364', u'heartbeat_front_addr': u'172.21.15.148:6813/27364', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27364, u'type': u'v2', u'addr': u'172.21.15.148:6808'}, {u'nonce': 27364, u'type': u'v1', u'addr': u'172.21.15.148:6809'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27364, u'type': u'v2', u'addr': u'172.21.15.148:6810'}, {u'nonce': 27364, u'type': u'v1', u'addr': u'172.21.15.148:6811'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.148:6809/27364', u'up_thru': 32, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27364, u'type': u'v2', u'addr': u'172.21.15.148:6814'}, {u'nonce': 27364, u'type': u'v1', u'addr': u'172.21.15.148:6815'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27364, u'type': u'v2', u'addr': u'172.21.15.148:6812'}, {u'nonce': 27364, u'type': u'v1', u'addr': u'172.21.15.148:6813'}]}, u'osd': 8}, {u'heartbeat_back_addr': u'172.21.15.50:6831/27643', u'uuid': u'd7ebf4f6-a2b8-4076-83dd-504172481e32', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.50:6827/27643', u'heartbeat_front_addr': u'172.21.15.50:6829/27643', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27643, u'type': u'v2', u'addr': u'172.21.15.50:6821'}, {u'nonce': 27643, u'type': u'v1', u'addr': u'172.21.15.50:6823'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27643, u'type': u'v2', u'addr': u'172.21.15.50:6825'}, {u'nonce': 27643, u'type': u'v1', u'addr': u'172.21.15.50:6827'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.50:6823/27643', u'up_thru': 48, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27643, u'type': u'v2', u'addr': u'172.21.15.50:6830'}, {u'nonce': 27643, u'type': u'v1', u'addr': u'172.21.15.50:6831'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27643, u'type': u'v2', u'addr': u'172.21.15.50:6828'}, {u'nonce': 27643, u'type': u'v1', u'addr': u'172.21.15.50:6829'}]}, u'osd': 9}, {u'heartbeat_back_addr': u'172.21.15.139:6825/26978', u'uuid': u'cb76df3e-6ea9-4ee6-8b03-84e3cd329c69', u'weight': 1, u'up_from': 23, u'cluster_addr': u'172.21.15.139:6814/26978', u'heartbeat_front_addr': u'172.21.15.139:6819/26978', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 26978, u'type': u'v2', u'addr': u'172.21.15.139:6808'}, {u'nonce': 26978, u'type': u'v1', u'addr': u'172.21.15.139:6810'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 26978, u'type': u'v2', u'addr': u'172.21.15.139:6812'}, {u'nonce': 26978, u'type': u'v1', u'addr': u'172.21.15.139:6814'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.139:6810/26978', u'up_thru': 53, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 26978, u'type': u'v2', u'addr': u'172.21.15.139:6822'}, {u'nonce': 26978, u'type': u'v1', u'addr': u'172.21.15.139:6825'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 26978, u'type': u'v2', u'addr': u'172.21.15.139:6816'}, {u'nonce': 26978, u'type': u'v1', u'addr': u'172.21.15.139:6819'}]}, u'osd': 10}, {u'heartbeat_back_addr': u'172.21.15.104:6817/28013', u'uuid': u'8bda27ba-cf17-472c-9a21-02dcaefab68f', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.104:6813/28013', u'heartbeat_front_addr': u'172.21.15.104:6815/28013', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 28013, u'type': u'v2', u'addr': u'172.21.15.104:6810'}, {u'nonce': 28013, u'type': u'v1', u'addr': u'172.21.15.104:6811'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 28013, u'type': u'v2', u'addr': u'172.21.15.104:6812'}, {u'nonce': 28013, u'type': u'v1', u'addr': u'172.21.15.104:6813'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.104:6811/28013', u'up_thru': 48, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 28013, u'type': u'v2', u'addr': u'172.21.15.104:6816'}, {u'nonce': 28013, u'type': u'v1', u'addr': u'172.21.15.104:6817'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 28013, u'type': u'v2', u'addr': u'172.21.15.104:6814'}, {u'nonce': 28013, u'type': u'v1', u'addr': u'172.21.15.104:6815'}]}, u'osd': 11}, {u'heartbeat_back_addr': u'172.21.15.148:6831/27395', u'uuid': u'd861da04-0be8-434b-9ed8-c1ec97fca7bd', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.148:6827/27395', u'heartbeat_front_addr': u'172.21.15.148:6829/27395', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27395, u'type': u'v2', u'addr': u'172.21.15.148:6823'}, {u'nonce': 27395, u'type': u'v1', u'addr': u'172.21.15.148:6825'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27395, u'type': u'v2', u'addr': u'172.21.15.148:6826'}, {u'nonce': 27395, u'type': u'v1', u'addr': u'172.21.15.148:6827'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.148:6825/27395', u'up_thru': 54, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27395, u'type': u'v2', u'addr': u'172.21.15.148:6830'}, {u'nonce': 27395, u'type': u'v1', u'addr': u'172.21.15.148:6831'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27395, u'type': u'v2', u'addr': u'172.21.15.148:6828'}, {u'nonce': 27395, u'type': u'v1', u'addr': u'172.21.15.148:6829'}]}, u'osd': 12}, {u'heartbeat_back_addr': u'172.21.15.50:6826/27631', u'uuid': u'4ba3410e-df5b-4529-9ebe-34c15f68bc9d', u'weight': 0, u'up_from': 24, u'cluster_addr': u'172.21.15.50:6819/27631', u'heartbeat_front_addr': u'172.21.15.50:6822/27631', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 27631, u'type': u'v2', u'addr': u'172.21.15.50:6816'}, {u'nonce': 27631, u'type': u'v1', u'addr': u'172.21.15.50:6817'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 27631, u'type': u'v2', u'addr': u'172.21.15.50:6818'}, {u'nonce': 27631, u'type': u'v1', u'addr': u'172.21.15.50:6819'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 0, u'public_addr': u'172.21.15.50:6817/27631', u'up_thru': 32, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 27631, u'type': u'v2', u'addr': u'172.21.15.50:6824'}, {u'nonce': 27631, u'type': u'v1', u'addr': u'172.21.15.50:6826'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 27631, u'type': u'v2', u'addr': u'172.21.15.50:6820'}, {u'nonce': 27631, u'type': u'v1', u'addr': u'172.21.15.50:6822'}]}, u'osd': 13}, {u'heartbeat_back_addr': u'172.21.15.139:6831/26981', u'uuid': u'1ba69402-6d5c-491e-ac86-747517533a8b', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.139:6827/26981', u'heartbeat_front_addr': u'172.21.15.139:6829/26981', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 26981, u'type': u'v2', u'addr': u'172.21.15.139:6818'}, {u'nonce': 26981, u'type': u'v1', u'addr': u'172.21.15.139:6821'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 26981, u'type': u'v2', u'addr': u'172.21.15.139:6824'}, {u'nonce': 26981, u'type': u'v1', u'addr': u'172.21.15.139:6827'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.139:6821/26981', u'up_thru': 54, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 26981, u'type': u'v2', u'addr': u'172.21.15.139:6830'}, {u'nonce': 26981, u'type': u'v1', u'addr': u'172.21.15.139:6831'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 26981, u'type': u'v2', u'addr': u'172.21.15.139:6828'}, {u'nonce': 26981, u'type': u'v1', u'addr': u'172.21.15.139:6829'}]}, u'osd': 14}, {u'heartbeat_back_addr': u'172.21.15.104:6833/28043', u'uuid': u'a3566380-0c4a-403f-8533-b2d0dcfa8506', u'weight': 1, u'up_from': 24, u'cluster_addr': u'172.21.15.104:6828/28043', u'heartbeat_front_addr': u'172.21.15.104:6831/28043', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 28043, u'type': u'v2', u'addr': u'172.21.15.104:6821'}, {u'nonce': 28043, u'type': u'v1', u'addr': u'172.21.15.104:6823'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 28043, u'type': u'v2', u'addr': u'172.21.15.104:6825'}, {u'nonce': 28043, u'type': u'v1', u'addr': u'172.21.15.104:6828'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.104:6823/28043', u'up_thru': 51, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 28043, u'type': u'v2', u'addr': u'172.21.15.104:6832'}, {u'nonce': 28043, u'type': u'v1', u'addr': u'172.21.15.104:6833'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 28043, u'type': u'v2', u'addr': u'172.21.15.104:6830'}, {u'nonce': 28043, u'type': u'v1', u'addr': u'172.21.15.104:6831'}]}, u'osd': 15}], u'crush_node_flags': {}, u'epoch': 56, u'pg_temp': [], u'require_min_compat_client': u'jewel', u'crush_version': 6, u'primary_temp': [], u'pool_max': 3, u'device_class_flags': {}, u'max_osd': 16, u'require_osd_release': u'octopus', u'min_compat_client': u'jewel', u'osd_xinfo': [{u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 0, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.085664+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 1, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.111969+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 2, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.060238+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 3, u'last_purged_snaps_scrub': u'2020-03-09T00:24:18.999353+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 4, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.026007+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 65536, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 5, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.046589+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 6, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.085042+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 7, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.111926+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 8, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.072146+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 9, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.062486+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 10, u'last_purged_snaps_scrub': u'2020-03-09T00:24:18.773795+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 11, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.033423+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 12, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.138144+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 65536, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 13, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.118617+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 14, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.039130+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 15, u'last_purged_snaps_scrub': u'2020-03-09T00:24:19.098144+0000'}], u'blacklist': {}, u'last_up_change': u'2020-03-09T00:24:20.875900+0000', u'flags_set': [u'noscrub', u'pglog_hardlimit', u'purged_snapdirs', u'recovery_deletes', u'sortbitwise'], u'fsid': u'036b76b6-511e-4ec6-9669-0a47550bd261', u'flags_num': 5801984, u'created': u'2020-03-09T00:24:06.736934+0000', u'modified': u'2020-03-09T00:25:05.792024+0000', u'flags': u'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', u'pg_upmap_items': [], u'pools': [{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 1, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:09.667386+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 1, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 1, u'pg_num': 1, u'type': 1, u'grade_table': [], u'pool_name': u'device_health_metrics', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 1, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'mgr_devicehealth': {}}, u'options': {u'pg_num_min': 1}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'26', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:26.419136+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'29', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 16384, u'flags_names': u'hashpspool,selfmanaged_snaps', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 16, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:40.793412+0000', u'snap_epoch': 56, u'erasure_code_profile': u'shecprofile', u'pg_autoscale_mode': u'off', u'snap_seq': 21, u'expected_num_objects': 0, u'size': 7, u'pg_num_pending': 16, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 16, u'pg_num': 16, u'type': 3, u'grade_table': [], u'pool_name': u'unique_pool_0', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 16, u'cache_mode': u'none', u'min_size': 5, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rados': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 8193, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'56', u'min_write_recency_for_promote': 0, u'read_tier': -1}], u'new_removed_snaps': [], u'removed_snaps_queue': [{u'snaps': [{u'begin': 4, u'length': 1}, {u'begin': 9, u'length': 1}, {u'begin': 11, u'length': 1}], u'pool': 3}]} 2020-03-09T00:25:06.373 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2020-03-09T00:25:06.373 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:06.368+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:06.373 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:25:06.470 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:06.469+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:06.510 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:06.511 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:06.521 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:25:06.571 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:06.570+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:06.646 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:06.646 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:06.655 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:25:06.698 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:06.697+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:06.763 INFO:tasks.rados.rados.0.smithi104.stdout:177: done (3 left) 2020-03-09T00:25:06.763 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 17 current snap is 14 2020-03-09T00:25:06.763 INFO:tasks.rados.rados.0.smithi104.stdout:178: seq_num 87 ranges {1818624=704512,2523136=688128,3211264=81920} 2020-03-09T00:25:06.765 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is set 2020-03-09T00:25:06.773 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:06.771+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:06.778 INFO:tasks.rados.rados.0.smithi104.stdout:178: writing smithi10428818-17 from 1818624 to 2523136 tid 1 2020-03-09T00:25:06.783 INFO:tasks.rados.rados.0.smithi104.stdout:178: writing smithi10428818-17 from 2523136 to 3211264 tid 2 2020-03-09T00:25:06.784 INFO:tasks.rados.rados.0.smithi104.stdout:178: writing smithi10428818-17 from 3211264 to 3293184 tid 3 2020-03-09T00:25:06.784 INFO:tasks.rados.rados.0.smithi104.stdout:174: finishing write tid 3 to smithi10428818-7 2020-03-09T00:25:06.785 INFO:tasks.rados.rados.0.smithi104.stdout:174: finishing write tid 5 to smithi10428818-7 2020-03-09T00:25:06.785 INFO:tasks.rados.rados.0.smithi104.stdout:175: finishing copy_from to smithi10428818-49 2020-03-09T00:25:06.785 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 49 v 19 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-03-09T00:25:06.785 INFO:tasks.rados.rados.0.smithi104.stdout:174: finishing write tid 6 to smithi10428818-7 2020-03-09T00:25:06.785 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 7 v 53 (ObjNum 86 snap 14 seq_num 86) dirty exists 2020-03-09T00:25:06.785 INFO:tasks.rados.rados.0.smithi104.stdout:174: left oid 7 (ObjNum 86 snap 14 seq_num 86) 2020-03-09T00:25:06.785 INFO:tasks.rados.rados.0.smithi104.stdout:175: finishing copy_from racing read to smithi10428818-49 2020-03-09T00:25:06.786 INFO:tasks.rados.rados.0.smithi104.stdout:176: expect (ObjNum 84 snap 12 seq_num 84) 2020-03-09T00:25:06.791 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:06.791 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:06.803 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:25:06.849 INFO:tasks.rados.rados.0.smithi104.stdout:178: finishing write tid 1 to smithi10428818-17 2020-03-09T00:25:06.849 INFO:tasks.rados.rados.0.smithi104.stdout:178: finishing write tid 2 to smithi10428818-17 2020-03-09T00:25:06.850 INFO:tasks.rados.rados.0.smithi104.stdout:178: finishing write tid 3 to smithi10428818-17 2020-03-09T00:25:06.850 INFO:tasks.rados.rados.0.smithi104.stdout:178: finishing write tid 5 to smithi10428818-17 2020-03-09T00:25:06.850 INFO:tasks.rados.rados.0.smithi104.stdout:178: finishing write tid 6 to smithi10428818-17 2020-03-09T00:25:06.850 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 17 v 26 (ObjNum 87 snap 14 seq_num 87) dirty exists 2020-03-09T00:25:06.850 INFO:tasks.rados.rados.0.smithi104.stdout:178: left oid 17 (ObjNum 87 snap 14 seq_num 87) 2020-03-09T00:25:06.850 INFO:tasks.rados.rados.0.smithi104.stdout:174: done (3 left) 2020-03-09T00:25:06.850 INFO:tasks.rados.rados.0.smithi104.stdout:175: done (2 left) 2020-03-09T00:25:06.851 INFO:tasks.rados.rados.0.smithi104.stdout:176: done (1 left) 2020-03-09T00:25:06.851 INFO:tasks.rados.rados.0.smithi104.stdout:178: done (0 left) 2020-03-09T00:25:06.851 INFO:tasks.rados.rados.0.smithi104.stdout:179: read oid 20 snap 9 2020-03-09T00:25:06.851 INFO:tasks.rados.rados.0.smithi104.stdout:179: expect (ObjNum 60 snap 6 seq_num 60) 2020-03-09T00:25:06.851 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 35 current snap is 14 2020-03-09T00:25:06.851 INFO:tasks.rados.rados.0.smithi104.stdout:180: seq_num 88 ranges {1998848=720896,2719744=442368,3162112=327680} 2020-03-09T00:25:06.864 INFO:tasks.rados.rados.0.smithi104.stdout:180: writing smithi10428818-35 from 1998848 to 2719744 tid 1 2020-03-09T00:25:06.866 INFO:tasks.rados.rados.0.smithi104.stdout:180: writing smithi10428818-35 from 2719744 to 3162112 tid 2 2020-03-09T00:25:06.869 INFO:tasks.rados.rados.0.smithi104.stdout:180: writing smithi10428818-35 from 3162112 to 3489792 tid 3 2020-03-09T00:25:06.869 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 39 current snap is 14 2020-03-09T00:25:06.869 INFO:tasks.rados.rados.0.smithi104.stdout:181: seq_num 89 ranges {638976=737280,1376256=573440,1949696=196608} 2020-03-09T00:25:06.873 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:06.872+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:06.876 INFO:tasks.rados.rados.0.smithi104.stdout:181: writing smithi10428818-39 from 638976 to 1376256 tid 1 2020-03-09T00:25:06.880 INFO:tasks.rados.rados.0.smithi104.stdout:181: writing smithi10428818-39 from 1376256 to 1949696 tid 2 2020-03-09T00:25:06.881 INFO:tasks.rados.rados.0.smithi104.stdout:181: writing smithi10428818-39 from 1949696 to 2146304 tid 3 2020-03-09T00:25:06.882 INFO:tasks.rados.rados.0.smithi104.stdout:182: copy_from oid 11 from oid 14 current snap is 14 2020-03-09T00:25:06.882 INFO:tasks.rados.rados.0.smithi104.stdout:179: expect (ObjNum 60 snap 6 seq_num 60) 2020-03-09T00:25:06.924 INFO:tasks.rados.rados.0.smithi104.stdout:180: finishing write tid 1 to smithi10428818-35 2020-03-09T00:25:06.924 INFO:tasks.rados.rados.0.smithi104.stdout:180: finishing write tid 2 to smithi10428818-35 2020-03-09T00:25:06.924 INFO:tasks.rados.rados.0.smithi104.stdout:180: finishing write tid 3 to smithi10428818-35 2020-03-09T00:25:06.925 INFO:tasks.rados.rados.0.smithi104.stdout:180: finishing write tid 5 to smithi10428818-35 2020-03-09T00:25:06.925 INFO:tasks.rados.rados.0.smithi104.stdout:180: finishing write tid 6 to smithi10428818-35 2020-03-09T00:25:06.925 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 35 v 30 (ObjNum 88 snap 14 seq_num 88) dirty exists 2020-03-09T00:25:06.925 INFO:tasks.rados.rados.0.smithi104.stdout:180: left oid 35 (ObjNum 88 snap 14 seq_num 88) 2020-03-09T00:25:06.925 INFO:tasks.rados.rados.0.smithi104.stdout:179: done (3 left) 2020-03-09T00:25:06.925 INFO:tasks.rados.rados.0.smithi104.stdout:180: done (2 left) 2020-03-09T00:25:06.925 INFO:tasks.rados.rados.0.smithi104.stdout:183: copy_from oid 34 from oid 48 current snap is 14 2020-03-09T00:25:06.926 INFO:tasks.rados.rados.0.smithi104.stdout:182: finishing copy_from to smithi10428818-11 2020-03-09T00:25:06.926 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 11 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-03-09T00:25:06.926 INFO:tasks.rados.rados.0.smithi104.stdout:181: finishing write tid 1 to smithi10428818-39 2020-03-09T00:25:06.926 INFO:tasks.rados.rados.0.smithi104.stdout:181: finishing write tid 2 to smithi10428818-39 2020-03-09T00:25:06.926 INFO:tasks.rados.rados.0.smithi104.stdout:181: finishing write tid 3 to smithi10428818-39 2020-03-09T00:25:06.926 INFO:tasks.rados.rados.0.smithi104.stdout:182: finishing copy_from racing read to smithi10428818-11 2020-03-09T00:25:06.926 INFO:tasks.rados.rados.0.smithi104.stdout:181: finishing write tid 5 to smithi10428818-39 2020-03-09T00:25:06.927 INFO:tasks.rados.rados.0.smithi104.stdout:181: finishing write tid 6 to smithi10428818-39 2020-03-09T00:25:06.927 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 39 v 54 (ObjNum 89 snap 14 seq_num 89) dirty exists 2020-03-09T00:25:06.927 INFO:tasks.rados.rados.0.smithi104.stdout:181: left oid 39 (ObjNum 89 snap 14 seq_num 89) 2020-03-09T00:25:06.927 INFO:tasks.rados.rados.0.smithi104.stdout:181: done (2 left) 2020-03-09T00:25:06.927 INFO:tasks.rados.rados.0.smithi104.stdout:182: done (1 left) 2020-03-09T00:25:06.927 INFO:tasks.rados.rados.0.smithi104.stdout:184: delete oid 6 current snap is 14 2020-03-09T00:25:06.928 INFO:tasks.rados.rados.0.smithi104.stdout:184: done (1 left) 2020-03-09T00:25:06.928 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 50 current snap is 14 2020-03-09T00:25:06.928 INFO:tasks.rados.rados.0.smithi104.stdout:185: seq_num 91 ranges {786432=770048,1556480=475136,2031616=294912} 2020-03-09T00:25:06.935 INFO:tasks.rados.rados.0.smithi104.stdout:185: writing smithi10428818-50 from 786432 to 1556480 tid 1 2020-03-09T00:25:06.938 INFO:tasks.rados.rados.0.smithi104.stdout:185: writing smithi10428818-50 from 1556480 to 2031616 tid 2 2020-03-09T00:25:06.940 INFO:tasks.rados.rados.0.smithi104.stdout:185: writing smithi10428818-50 from 2031616 to 2326528 tid 3 2020-03-09T00:25:06.940 INFO:tasks.rados.rados.0.smithi104.stdout:186: snap_remove snap 10 2020-03-09T00:25:06.958 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:06.958 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:06.968 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:25:06.974 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:06.973+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:07.075 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:07.074+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:07.085 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:07.086 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:07.094 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:25:07.095 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:07.176 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:07.174+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:07.214 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:07.215 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:07.224 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:25:07.230 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:07.251 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:07.272 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:07.277 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:07.276+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:07.292 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:07.311 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:07.331 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:07.341 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:07.341 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:07.349 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:25:07.351 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:25:07.371 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:07.390 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:07.393 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:07.392+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:07.410 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:07.430 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:07.451 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:07.465 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:07.465 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:07.465 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:07.465 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:07.465 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:07.470 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:07.473 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:25:07.490 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:07.510 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:07.512 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:07.510+0000 7fe98a637700 0 -- 172.21.15.148:0/4193448672 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7fe98418b550 msgr2=0x7fe98418d940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:07.514 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:07.513+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:07.532 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.534 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.535 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.536 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.536 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.536 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.536 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.536 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.537 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.537 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:07.580 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:07.579+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:07.630 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:07.631 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.800649468, 2020-03-09T00:25:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737, 2020-03-09T00:25:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000156164 2020-03-09T00:25:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827665+0000", 2020-03-09T00:25:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5397000000000001e-05 2020-03-09T00:25:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827690+0000", 2020-03-09T00:25:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6900000000000002e-05 2020-03-09T00:25:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827727+0000", 2020-03-09T00:25:07.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9529999999999999e-06 2020-03-09T00:25:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827735+0000", 2020-03-09T00:25:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011533660000000001 2020-03-09T00:25:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828889+0000", 2020-03-09T00:25:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1542999999999999e-05 2020-03-09T00:25:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828900+0000", 2020-03-09T00:25:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038471730000000002 2020-03-09T00:25:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832747+0000", 2020-03-09T00:25:07.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8874000000000001e-05 2020-03-09T00:25:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832786+0000", 2020-03-09T00:25:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737 2020-03-09T00:25:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.639 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/32 ECSubWrite(tid=11, reqid=client.4751.0:197, at_version=38'7, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.832737+0000", 2020-03-09T00:25:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.795421591, 2020-03-09T00:25:07.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028425999999999998, 2020-03-09T00:25:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832737+0000", 2020-03-09T00:25:07.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832737+0000", 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0211999999999996e-05 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832827+0000", 2020-03-09T00:25:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8890999999999998e-05 2020-03-09T00:25:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832856+0000", 2020-03-09T00:25:07.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5639e-05 2020-03-09T00:25:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832881+0000", 2020-03-09T00:25:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0940000000000002e-06 2020-03-09T00:25:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832889+0000", 2020-03-09T00:25:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7373999999999997e-05 2020-03-09T00:25:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832936+0000", 2020-03-09T00:25:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4147e-05 2020-03-09T00:25:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832950+0000", 2020-03-09T00:25:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025816960000000001 2020-03-09T00:25:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835532+0000", 2020-03-09T00:25:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7546999999999997e-05 2020-03-09T00:25:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835579+0000", 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0028425999999999998 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:25:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.764052039999999, 2020-03-09T00:25:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045310819999999996, 2020-03-09T00:25:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.649 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000157849 2020-03-09T00:25:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864264+0000", 2020-03-09T00:25:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8293000000000002e-05 2020-03-09T00:25:07.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864292+0000", 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6464e-05 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864309+0000", 2020-03-09T00:25:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9360000000000002e-06 2020-03-09T00:25:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864316+0000", 2020-03-09T00:25:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3850000000000001e-05 2020-03-09T00:25:07.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864370+0000", 2020-03-09T00:25:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3759e-05 2020-03-09T00:25:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864383+0000", 2020-03-09T00:25:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041983749999999999 2020-03-09T00:25:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868582+0000", 2020-03-09T00:25:07.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5556000000000001e-05 2020-03-09T00:25:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868637+0000", 2020-03-09T00:25:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045310819999999996 2020-03-09T00:25:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 38/32 ECSubWrite(tid=6, reqid=client.4751.0:207, at_version=0'0, trim_to=0'0, roll_forward_to=38'4))", 2020-03-09T00:25:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.870474+0000", 2020-03-09T00:25:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.757684510000001, 2020-03-09T00:25:07.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001860013, 2020-03-09T00:25:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870474+0000", 2020-03-09T00:25:07.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870474+0000", 2020-03-09T00:25:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9252999999999997e-05 2020-03-09T00:25:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870533+0000", 2020-03-09T00:25:07.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5296999999999999e-05 2020-03-09T00:25:07.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870548+0000", 2020-03-09T00:25:07.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4623999999999999e-05 2020-03-09T00:25:07.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870563+0000", 2020-03-09T00:25:07.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5739999999999998e-06 2020-03-09T00:25:07.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870571+0000", 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00127352 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871844+0000", 2020-03-09T00:25:07.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3760999999999999e-05 2020-03-09T00:25:07.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871858+0000", 2020-03-09T00:25:07.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00044278999999999998 2020-03-09T00:25:07.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.872301+0000", 2020-03-09T00:25:07.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3194000000000001e-05 2020-03-09T00:25:07.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.872334+0000", 2020-03-09T00:25:07.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001860013 2020-03-09T00:25:07.665 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.665 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.665 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 39/32 ECSubWrite(tid=22, reqid=client.4751.0:210, at_version=39'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:48.828524+0000", 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.799634046000001, 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037733570000000002, 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.666 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.667 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828524+0000", 2020-03-09T00:25:07.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828524+0000", 2020-03-09T00:25:07.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00022246500000000001 2020-03-09T00:25:07.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828747+0000", 2020-03-09T00:25:07.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7589999999999998e-05 2020-03-09T00:25:07.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828784+0000", 2020-03-09T00:25:07.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6183e-05 2020-03-09T00:25:07.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828811+0000", 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0047000000000001e-05 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828821+0000", 2020-03-09T00:25:07.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5307000000000003e-05 2020-03-09T00:25:07.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828886+0000", 2020-03-09T00:25:07.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1233000000000001e-05 2020-03-09T00:25:07.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.828897+0000", 2020-03-09T00:25:07.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033575749999999998 2020-03-09T00:25:07.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.832255+0000", 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2957000000000002e-05 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:48.832298+0000", 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037733570000000002 2020-03-09T00:25:07.673 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.674 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.674 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.674 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 43/32 ECSubWrite(tid=27, reqid=client.4751.0:237, at_version=43'10, trim_to=0'0, roll_forward_to=39'8))", 2020-03-09T00:25:07.674 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:52.849039+0000", 2020-03-09T00:25:07.674 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 14.779119466999999, 2020-03-09T00:25:07.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003813486, 2020-03-09T00:25:07.675 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.675 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.675 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849039+0000", 2020-03-09T00:25:07.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849039+0000", 2020-03-09T00:25:07.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000236736 2020-03-09T00:25:07.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849276+0000", 2020-03-09T00:25:07.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8025e-05 2020-03-09T00:25:07.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849304+0000", 2020-03-09T00:25:07.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4955e-05 2020-03-09T00:25:07.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.678 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849319+0000", 2020-03-09T00:25:07.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3359999999999995e-06 2020-03-09T00:25:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849327+0000", 2020-03-09T00:25:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1693999999999994e-05 2020-03-09T00:25:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849409+0000", 2020-03-09T00:25:07.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5019999999999998e-06 2020-03-09T00:25:07.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.849418+0000", 2020-03-09T00:25:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033891720000000002 2020-03-09T00:25:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.852807+0000", 2020-03-09T00:25:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5065999999999998e-05 2020-03-09T00:25:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.852852+0000", 2020-03-09T00:25:07.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003813486 2020-03-09T00:25:07.682 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.682 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.682 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.683 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 43/32 ECSubWrite(tid=28, reqid=client.4751.0:238, at_version=43'11, trim_to=0'0, roll_forward_to=39'8))", 2020-03-09T00:25:07.683 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:52.853909+0000", 2020-03-09T00:25:07.683 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 14.77424931, 2020-03-09T00:25:07.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021767790000000002, 2020-03-09T00:25:07.683 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.684 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.684 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.853909+0000", 2020-03-09T00:25:07.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.853909+0000", 2020-03-09T00:25:07.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020812000000000001 2020-03-09T00:25:07.685 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.854117+0000", 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1257999999999999e-05 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.854138+0000", 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3835999999999999e-05 2020-03-09T00:25:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.854152+0000", 2020-03-09T00:25:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5759999999999995e-06 2020-03-09T00:25:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.854162+0000", 2020-03-09T00:25:07.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1480000000000003e-05 2020-03-09T00:25:07.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.688 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.688 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.688 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.854203+0000", 2020-03-09T00:25:07.688 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.306e-06 2020-03-09T00:25:07.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.689 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.689 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.854211+0000", 2020-03-09T00:25:07.689 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001835408 2020-03-09T00:25:07.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.689 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.856046+0000", 2020-03-09T00:25:07.690 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9795000000000001e-05 2020-03-09T00:25:07.690 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.690 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.690 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.690 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:52.856086+0000", 2020-03-09T00:25:07.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021767790000000002 2020-03-09T00:25:07.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.691 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.691 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.691 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 44/32 ECSubWrite(tid=33, reqid=osd.10.0:32, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2020-03-09T00:25:07.692 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.857743+0000", 2020-03-09T00:25:07.692 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.770415583, 2020-03-09T00:25:07.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001888827, 2020-03-09T00:25:07.692 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.692 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.692 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857743+0000", 2020-03-09T00:25:07.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857743+0000", 2020-03-09T00:25:07.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3237000000000005e-05 2020-03-09T00:25:07.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.694 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857806+0000", 2020-03-09T00:25:07.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4936000000000001e-05 2020-03-09T00:25:07.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857821+0000", 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5472e-05 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.695 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857836+0000", 2020-03-09T00:25:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1300000000000001e-06 2020-03-09T00:25:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.857845+0000", 2020-03-09T00:25:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011418310000000001 2020-03-09T00:25:07.696 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.858986+0000", 2020-03-09T00:25:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2933e-05 2020-03-09T00:25:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.697 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.858999+0000", 2020-03-09T00:25:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000584765 2020-03-09T00:25:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859584+0000", 2020-03-09T00:25:07.698 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7522999999999999e-05 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859632+0000", 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001888827 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.699 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 44/32 ECSubWrite(tid=34, reqid=client.4751.0:249, at_version=44'16, trim_to=0'0, roll_forward_to=44'14))", 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.867338+0000", 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.760820646000001, 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038215200000000001, 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.700 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867338+0000", 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867338+0000", 2020-03-09T00:25:07.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027359900000000002 2020-03-09T00:25:07.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867611+0000", 2020-03-09T00:25:07.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4257000000000001e-05 2020-03-09T00:25:07.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867646+0000", 2020-03-09T00:25:07.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1376e-05 2020-03-09T00:25:07.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867667+0000", 2020-03-09T00:25:07.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1700000000000003e-06 2020-03-09T00:25:07.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867676+0000", 2020-03-09T00:25:07.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2354999999999995e-05 2020-03-09T00:25:07.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867738+0000", 2020-03-09T00:25:07.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6375000000000002e-05 2020-03-09T00:25:07.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867755+0000", 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033428120000000001 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.706 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.871098+0000", 2020-03-09T00:25:07.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1575999999999995e-05 2020-03-09T00:25:07.707 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.707 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.707 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.707 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.871159+0000", 2020-03-09T00:25:07.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038215200000000001 2020-03-09T00:25:07.707 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 48/32 ECSubWrite(tid=38, reqid=osd.10.0:38, at_version=48'18, trim_to=0'0, roll_forward_to=44'16))", 2020-03-09T00:25:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:57.771661+0000", 2020-03-09T00:25:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.8564970600000006, 2020-03-09T00:25:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016075869999999999, 2020-03-09T00:25:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.771661+0000", 2020-03-09T00:25:07.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.771661+0000", 2020-03-09T00:25:07.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7156e-05 2020-03-09T00:25:07.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.771718+0000", 2020-03-09T00:25:07.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4414e-05 2020-03-09T00:25:07.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.771743+0000", 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7153e-05 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.771760+0000", 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0589999999999997e-06 2020-03-09T00:25:07.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.771767+0000", 2020-03-09T00:25:07.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00054075399999999995 2020-03-09T00:25:07.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.772308+0000", 2020-03-09T00:25:07.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2812e-05 2020-03-09T00:25:07.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.772321+0000", 2020-03-09T00:25:07.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00090231999999999997 2020-03-09T00:25:07.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.773223+0000", 2020-03-09T00:25:07.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5918999999999999e-05 2020-03-09T00:25:07.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:57.773269+0000", 2020-03-09T00:25:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016075869999999999 2020-03-09T00:25:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 49/32 ECSubWrite(tid=41, reqid=client.4751.0:279, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.794656+0000", 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.8335022950000006, 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030702289999999998, 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.717 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.794656+0000", 2020-03-09T00:25:07.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.794656+0000", 2020-03-09T00:25:07.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016890199999999999 2020-03-09T00:25:07.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.794825+0000", 2020-03-09T00:25:07.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0477e-05 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.794855+0000", 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5781999999999999e-05 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.794871+0000", 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3200000000000002e-06 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.794878+0000", 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000126317 2020-03-09T00:25:07.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.795005+0000", 2020-03-09T00:25:07.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3599e-05 2020-03-09T00:25:07.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.795018+0000", 2020-03-09T00:25:07.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026561699999999998 2020-03-09T00:25:07.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.797675+0000", 2020-03-09T00:25:07.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1662000000000001e-05 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.797726+0000", 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0030702289999999998 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.724 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.725 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s1 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:07.725 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:07.725 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.8353710750000003, 2020-03-09T00:25:07.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997, 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062438599999999999 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793412+0000", 2020-03-09T00:25:07.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6639e-05 2020-03-09T00:25:07.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793438+0000", 2020-03-09T00:25:07.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3116999999999999e-05 2020-03-09T00:25:07.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793461+0000", 2020-03-09T00:25:07.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2469999999999997e-06 2020-03-09T00:25:07.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793471+0000", 2020-03-09T00:25:07.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9004999999999994e-05 2020-03-09T00:25:07.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.730 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.730 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793540+0000", 2020-03-09T00:25:07.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8408000000000001e-05 2020-03-09T00:25:07.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.731 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.731 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793558+0000", 2020-03-09T00:25:07.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053623589999999997 2020-03-09T00:25:07.731 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798920+0000", 2020-03-09T00:25:07.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2633000000000002e-05 2020-03-09T00:25:07.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798973+0000", 2020-03-09T00:25:07.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997 2020-03-09T00:25:07.733 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.733 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.733 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.733 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s1 53/52 ECSubWrite(tid=2, reqid=client.4751.0:328, at_version=53'17, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:07.734 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.801781+0000", 2020-03-09T00:25:07.734 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.8263774469999996, 2020-03-09T00:25:07.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031942160000000002, 2020-03-09T00:25:07.734 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.734 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.734 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.801781+0000", 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.801781+0000", 2020-03-09T00:25:07.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038706800000000001 2020-03-09T00:25:07.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802168+0000", 2020-03-09T00:25:07.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6656e-05 2020-03-09T00:25:07.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802195+0000", 2020-03-09T00:25:07.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5906e-05 2020-03-09T00:25:07.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802210+0000", 2020-03-09T00:25:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9290000000000001e-06 2020-03-09T00:25:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802217+0000", 2020-03-09T00:25:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5695999999999998e-05 2020-03-09T00:25:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802263+0000", 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3978999999999999e-05 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.739 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802277+0000", 2020-03-09T00:25:07.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026468199999999998 2020-03-09T00:25:07.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.804924+0000", 2020-03-09T00:25:07.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1162000000000002e-05 2020-03-09T00:25:07.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.804975+0000", 2020-03-09T00:25:07.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031942160000000002 2020-03-09T00:25:07.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.741 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.741 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.742 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=30, reqid=client.4751.0:346, at_version=53'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:07.742 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:07.742 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.8011130130000002, 2020-03-09T00:25:07.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999, 2020-03-09T00:25:07.742 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013296799999999999 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827178+0000", 2020-03-09T00:25:07.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6305e-05 2020-03-09T00:25:07.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827205+0000", 2020-03-09T00:25:07.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5832999999999999e-05 2020-03-09T00:25:07.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827220+0000", 2020-03-09T00:25:07.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7170000000000008e-06 2020-03-09T00:25:07.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827228+0000", 2020-03-09T00:25:07.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2020-03-09T00:25:07.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.747 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.747 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827292+0000", 2020-03-09T00:25:07.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4157e-05 2020-03-09T00:25:07.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827306+0000", 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046893439999999998 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831995+0000", 2020-03-09T00:25:07.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3732000000000006e-05 2020-03-09T00:25:07.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832059+0000", 2020-03-09T00:25:07.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999 2020-03-09T00:25:07.749 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.750 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.750 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.750 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=31, reqid=client.4751.0:347, at_version=53'19, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:07.750 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:07.751 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7932748729999997, 2020-03-09T00:25:07.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996, 2020-03-09T00:25:07.753 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.753 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.753 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:07.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:07.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030527400000000002 2020-03-09T00:25:07.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835189+0000", 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4273999999999998e-05 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835213+0000", 2020-03-09T00:25:07.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7130999999999999e-05 2020-03-09T00:25:07.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.756 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.756 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835230+0000", 2020-03-09T00:25:07.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6400000000000003e-06 2020-03-09T00:25:07.756 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835239+0000", 2020-03-09T00:25:07.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0665000000000003e-05 2020-03-09T00:25:07.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835299+0000", 2020-03-09T00:25:07.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6062e-05 2020-03-09T00:25:07.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835315+0000", 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005161349 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840477+0000", 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7466000000000003e-05 2020-03-09T00:25:07.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840524+0000", 2020-03-09T00:25:07.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996 2020-03-09T00:25:07.761 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.761 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.761 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.762 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=34, reqid=client.4751.0:349, at_version=0'0, trim_to=0'0, roll_forward_to=53'21))", 2020-03-09T00:25:07.762 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:07.762 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7826968230000002, 2020-03-09T00:25:07.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001, 2020-03-09T00:25:07.762 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:07.764 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6162000000000002e-05 2020-03-09T00:25:07.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.764 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.764 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845518+0000", 2020-03-09T00:25:07.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3681e-05 2020-03-09T00:25:07.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845531+0000", 2020-03-09T00:25:07.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376e-05 2020-03-09T00:25:07.765 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845547+0000", 2020-03-09T00:25:07.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8040000000000002e-06 2020-03-09T00:25:07.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845555+0000", 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010135 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845656+0000", 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2714000000000001e-05 2020-03-09T00:25:07.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845669+0000", 2020-03-09T00:25:07.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072973140000000001 2020-03-09T00:25:07.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.852966+0000", 2020-03-09T00:25:07.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2704999999999998e-05 2020-03-09T00:25:07.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.853019+0000", 2020-03-09T00:25:07.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001 2020-03-09T00:25:07.770 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.770 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.770 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.770 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 53/32 ECSubWrite(tid=57, reqid=client.4751.0:362, at_version=53'28, trim_to=0'0, roll_forward_to=52'26))", 2020-03-09T00:25:07.770 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.893836+0000", 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7343220559999999, 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033220390000000002, 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.893836+0000", 2020-03-09T00:25:07.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.772 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.772 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.893836+0000", 2020-03-09T00:25:07.772 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016310599999999999 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.893999+0000", 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7937000000000001e-05 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.773 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.894017+0000", 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0380000000000001e-05 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.894028+0000", 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.387e-06 2020-03-09T00:25:07.774 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.894034+0000", 2020-03-09T00:25:07.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8248999999999998e-05 2020-03-09T00:25:07.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.894082+0000", 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7429999999999997e-06 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.894092+0000", 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00303272 2020-03-09T00:25:07.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.897125+0000", 2020-03-09T00:25:07.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3516999999999998e-05 2020-03-09T00:25:07.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.897158+0000", 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033220390000000002 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.778 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 53/32 ECSubWrite(tid=58, reqid=client.4751.0:363, at_version=53'29, trim_to=0'0, roll_forward_to=52'26))", 2020-03-09T00:25:07.779 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.899173+0000", 2020-03-09T00:25:07.779 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.728985003, 2020-03-09T00:25:07.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002366319, 2020-03-09T00:25:07.779 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.779 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.779 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899173+0000", 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899173+0000", 2020-03-09T00:25:07.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012762200000000001 2020-03-09T00:25:07.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899301+0000", 2020-03-09T00:25:07.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4609e-05 2020-03-09T00:25:07.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899316+0000", 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1755e-05 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899327+0000", 2020-03-09T00:25:07.782 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3550000000000004e-06 2020-03-09T00:25:07.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899333+0000", 2020-03-09T00:25:07.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4662e-05 2020-03-09T00:25:07.783 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899377+0000", 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2244e-05 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.899390+0000", 2020-03-09T00:25:07.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021193900000000001 2020-03-09T00:25:07.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.901509+0000", 2020-03-09T00:25:07.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0682000000000002e-05 2020-03-09T00:25:07.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.901540+0000", 2020-03-09T00:25:07.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002366319 2020-03-09T00:25:07.787 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.787 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.787 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.789 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 53/32 ECSubWrite(tid=59, reqid=client.4751.0:364, at_version=53'30, trim_to=0'0, roll_forward_to=52'26))", 2020-03-09T00:25:07.789 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.903048+0000", 2020-03-09T00:25:07.789 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7251104269999997, 2020-03-09T00:25:07.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015334669999999999, 2020-03-09T00:25:07.790 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.790 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.790 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.790 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903048+0000", 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903048+0000", 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103564 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903151+0000", 2020-03-09T00:25:07.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8445e-05 2020-03-09T00:25:07.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903170+0000", 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2825e-05 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903183+0000", 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.939e-06 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903190+0000", 2020-03-09T00:25:07.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6916000000000002e-05 2020-03-09T00:25:07.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903247+0000", 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4869999999999999e-06 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.903253+0000", 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012975009999999999 2020-03-09T00:25:07.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.904551+0000", 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0790000000000002e-05 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.904581+0000", 2020-03-09T00:25:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015334669999999999 2020-03-09T00:25:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 53/32 ECSubWrite(tid=61, reqid=client.4751.0:365, at_version=0'0, trim_to=0'0, roll_forward_to=53'31))", 2020-03-09T00:25:07.798 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.907777+0000", 2020-03-09T00:25:07.798 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.7203818049999997, 2020-03-09T00:25:07.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033291839999999998, 2020-03-09T00:25:07.798 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:07.798 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:07.798 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:07.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907777+0000", 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907777+0000", 2020-03-09T00:25:07.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6940999999999995e-05 2020-03-09T00:25:07.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:07.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907843+0000", 2020-03-09T00:25:07.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4398e-05 2020-03-09T00:25:07.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:07.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907858+0000", 2020-03-09T00:25:07.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9029e-05 2020-03-09T00:25:07.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:07.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907877+0000", 2020-03-09T00:25:07.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9000000000000006e-06 2020-03-09T00:25:07.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907885+0000", 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7670000000000003e-05 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:07.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907932+0000", 2020-03-09T00:25:07.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3499999999999999e-05 2020-03-09T00:25:07.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:07.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.907946+0000", 2020-03-09T00:25:07.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003129822 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.911076+0000", 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9924e-05 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:07.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.911106+0000", 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033291839999999998 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:07.806 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:07.808 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-03-09T00:25:07.809 INFO:tasks.rados.rados.0.smithi104.stdout:186: done (2 left) 2020-03-09T00:25:07.809 INFO:tasks.rados.rados.0.smithi104.stdout:187: snap_remove snap 9 2020-03-09T00:25:07.810 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:07.807+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:07.908 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:07.907+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:07.949 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:07.949 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:07.950 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:07.950 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:07.958 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-03-09T00:25:08.010 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:08.008+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:08.073 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:08.073 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:08.074 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:08.074 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:08.074 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:08.081 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-03-09T00:25:08.110 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:08.109+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:08.199 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:08.199 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:08.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:08.199 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:354 3.1s0 3:8a6d1775:::smithi10428818-12:head [append 0~491520] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.841900+0000", 2020-03-09T00:25:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3543593429999996, 2020-03-09T00:25:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018706021, 2020-03-09T00:25:08.200 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 354 2020-03-09T00:25:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.201 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.841900+0000", 2020-03-09T00:25:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.841900+0000", 2020-03-09T00:25:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001368537 2020-03-09T00:25:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.843268+0000", 2020-03-09T00:25:08.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.0610000000000002e-06 2020-03-09T00:25:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.843277+0000", 2020-03-09T00:25:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4048e-05 2020-03-09T00:25:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.843292+0000", 2020-03-09T00:25:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5270000000000004e-06 2020-03-09T00:25:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.843298+0000", 2020-03-09T00:25:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0735000000000006e-05 2020-03-09T00:25:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.843369+0000", 2020-03-09T00:25:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00034074300000000001 2020-03-09T00:25:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.843710+0000", 2020-03-09T00:25:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00191628 2020-03-09T00:25:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.845626+0000", 2020-03-09T00:25:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014323822999999999 2020-03-09T00:25:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.859950+0000", 2020-03-09T00:25:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014676399999999999 2020-03-09T00:25:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.860096+0000", 2020-03-09T00:25:08.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00050950300000000002 2020-03-09T00:25:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.860606+0000", 2020-03-09T00:25:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018706021 2020-03-09T00:25:08.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:355 3.1s0 3:8a6d1775:::smithi10428818-12:head [append 0~770048] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:08.212 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.846780+0000", 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3494787319999997, 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016858669, 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.213 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 355 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.846780+0000", 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.846780+0000", 2020-03-09T00:25:08.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011107090000000001 2020-03-09T00:25:08.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.847891+0000", 2020-03-09T00:25:08.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1289999999999996e-06 2020-03-09T00:25:08.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.847899+0000", 2020-03-09T00:25:08.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3555e-05 2020-03-09T00:25:08.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.847913+0000", 2020-03-09T00:25:08.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.122e-06 2020-03-09T00:25:08.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.847919+0000", 2020-03-09T00:25:08.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001895781 2020-03-09T00:25:08.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.849815+0000", 2020-03-09T00:25:08.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000206808 2020-03-09T00:25:08.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850022+0000", 2020-03-09T00:25:08.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001259727 2020-03-09T00:25:08.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851281+0000", 2020-03-09T00:25:08.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096055640000000005 2020-03-09T00:25:08.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.860887+0000", 2020-03-09T00:25:08.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002393811 2020-03-09T00:25:08.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.863281+0000", 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000358463 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.863639+0000", 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016858669 2020-03-09T00:25:08.223 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.224 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.224 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:356 3.1s0 3:8a6d1775:::smithi10428818-12:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:08.224 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.847983+0000", 2020-03-09T00:25:08.224 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3482763670000004, 2020-03-09T00:25:08.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016514021, 2020-03-09T00:25:08.225 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.225 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.225 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.225 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.225 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.226 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 356 2020-03-09T00:25:08.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.226 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.847983+0000", 2020-03-09T00:25:08.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.227 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.847983+0000", 2020-03-09T00:25:08.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.2942000000000006e-05 2020-03-09T00:25:08.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848066+0000", 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0979999999999994e-06 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848074+0000", 2020-03-09T00:25:08.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2520999999999999e-05 2020-03-09T00:25:08.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848086+0000", 2020-03-09T00:25:08.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5259999999999999e-06 2020-03-09T00:25:08.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848093+0000", 2020-03-09T00:25:08.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0093732599999999996 2020-03-09T00:25:08.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.857466+0000", 2020-03-09T00:25:08.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00028267599999999999 2020-03-09T00:25:08.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.857749+0000", 2020-03-09T00:25:08.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010570309999999999 2020-03-09T00:25:08.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.858806+0000", 2020-03-09T00:25:08.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0020822390000000001 2020-03-09T00:25:08.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.860888+0000", 2020-03-09T00:25:08.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003012158 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.863900+0000", 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00059657000000000004 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.864497+0000", 2020-03-09T00:25:08.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016514021 2020-03-09T00:25:08.235 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.235 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.235 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:357 3.1s0 3:8a6d1775:::smithi10428818-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.848139+0000", 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3481202440000004, 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020774082999999999, 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:08.236 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.237 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.237 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.237 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 357 2020-03-09T00:25:08.237 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.237 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848139+0000", 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848139+0000", 2020-03-09T00:25:08.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4647e-05 2020-03-09T00:25:08.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848164+0000", 2020-03-09T00:25:08.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.7179999999999998e-06 2020-03-09T00:25:08.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848167+0000", 2020-03-09T00:25:08.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.401e-06 2020-03-09T00:25:08.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848175+0000", 2020-03-09T00:25:08.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8569999999999997e-06 2020-03-09T00:25:08.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.848179+0000", 2020-03-09T00:25:08.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011499809 2020-03-09T00:25:08.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.859678+0000", 2020-03-09T00:25:08.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026053300000000002 2020-03-09T00:25:08.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:08.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.859939+0000", 2020-03-09T00:25:08.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.005264197 2020-03-09T00:25:08.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.865203+0000", 2020-03-09T00:25:08.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023974399999999999 2020-03-09T00:25:08.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.865443+0000", 2020-03-09T00:25:08.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0034701770000000001 2020-03-09T00:25:08.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.868913+0000", 2020-03-09T00:25:08.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020774082999999999 2020-03-09T00:25:08.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.246 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.246 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.246 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.247 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 53/52 ECSubWriteReply(tid=46, last_complete=53'22, committed=1, applied=1))", 2020-03-09T00:25:08.247 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.849966+0000", 2020-03-09T00:25:08.247 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.3462929969999999, 2020-03-09T00:25:08.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0091760560000000001, 2020-03-09T00:25:08.247 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.247 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.849966+0000", 2020-03-09T00:25:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.849966+0000", 2020-03-09T00:25:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8651999999999999e-05 2020-03-09T00:25:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.849995+0000", 2020-03-09T00:25:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.955e-06 2020-03-09T00:25:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850000+0000", 2020-03-09T00:25:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2173e-05 2020-03-09T00:25:08.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850012+0000", 2020-03-09T00:25:08.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2529999999999998e-06 2020-03-09T00:25:08.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850016+0000", 2020-03-09T00:25:08.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0091044339999999998 2020-03-09T00:25:08.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.859121+0000", 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1588999999999999e-05 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.859142+0000", 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0091760560000000001 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.253 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:402 3.1s0 3:81f9c489:::smithi10428818-29:head [append 0~589824] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.839127+0000", 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3571324110000003, 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99676392999999996, 2020-03-09T00:25:08.254 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.255 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.255 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.255 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.255 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.255 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 402 2020-03-09T00:25:08.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.255 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.256 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.256 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.839127+0000", 2020-03-09T00:25:08.256 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.257 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.257 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.839127+0000", 2020-03-09T00:25:08.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010327839999999999 2020-03-09T00:25:08.257 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.257 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.257 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.840160+0000", 2020-03-09T00:25:08.258 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0879999999999995e-06 2020-03-09T00:25:08.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.258 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.840168+0000", 2020-03-09T00:25:08.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1980999999999999e-05 2020-03-09T00:25:08.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.840180+0000", 2020-03-09T00:25:08.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6939999999999998e-06 2020-03-09T00:25:08.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.840186+0000", 2020-03-09T00:25:08.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2012000000000002e-05 2020-03-09T00:25:08.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.840248+0000", 2020-03-09T00:25:08.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.439e-06 2020-03-09T00:25:08.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:08.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.840252+0000", 2020-03-09T00:25:08.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.96515789699999999 2020-03-09T00:25:08.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.805410+0000", 2020-03-09T00:25:08.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015191900000000001 2020-03-09T00:25:08.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.805562+0000", 2020-03-09T00:25:08.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00137947 2020-03-09T00:25:08.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.806941+0000", 2020-03-09T00:25:08.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016695061000000001 2020-03-09T00:25:08.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823636+0000", 2020-03-09T00:25:08.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011831753 2020-03-09T00:25:08.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.835468+0000", 2020-03-09T00:25:08.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00042283199999999999 2020-03-09T00:25:08.266 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.266 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.266 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.266 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.835891+0000", 2020-03-09T00:25:08.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99676392999999996 2020-03-09T00:25:08.266 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.267 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.267 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.267 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.267 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:403 3.1s0 3:81f9c489:::smithi10428818-29:head [append 0~573440] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.267 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.842218+0000", 2020-03-09T00:25:08.268 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3540416139999998, 2020-03-09T00:25:08.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99391352799999999, 2020-03-09T00:25:08.268 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.268 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.268 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.268 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.268 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 403 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.842218+0000", 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.269 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.270 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.842218+0000", 2020-03-09T00:25:08.270 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0018191049999999999 2020-03-09T00:25:08.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.270 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.270 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844037+0000", 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.489e-06 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844043+0000", 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3213999999999999e-05 2020-03-09T00:25:08.271 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.272 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.272 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844056+0000", 2020-03-09T00:25:08.272 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9650000000000002e-06 2020-03-09T00:25:08.272 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.272 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844063+0000", 2020-03-09T00:25:08.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8072000000000002e-05 2020-03-09T00:25:08.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844121+0000", 2020-03-09T00:25:08.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2480000000000001e-06 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844125+0000", 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.96868389799999999 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812809+0000", 2020-03-09T00:25:08.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000165007 2020-03-09T00:25:08.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.275 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812974+0000", 2020-03-09T00:25:08.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001383034 2020-03-09T00:25:08.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.276 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.814357+0000", 2020-03-09T00:25:08.276 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0092821259999999999 2020-03-09T00:25:08.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823639+0000", 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012405810999999999 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.836045+0000", 2020-03-09T00:25:08.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6558999999999996e-05 2020-03-09T00:25:08.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.278 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.836131+0000", 2020-03-09T00:25:08.278 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99391352799999999 2020-03-09T00:25:08.278 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.278 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:404 3.1s0 3:81f9c489:::smithi10428818-29:head [append 0~196608] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.844103+0000", 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3521562280000001, 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99754793900000005, 2020-03-09T00:25:08.279 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 404 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844103+0000", 2020-03-09T00:25:08.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844103+0000", 2020-03-09T00:25:08.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00076300300000000005 2020-03-09T00:25:08.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.282 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.282 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844866+0000", 2020-03-09T00:25:08.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8269999999999997e-06 2020-03-09T00:25:08.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.284 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.284 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844871+0000", 2020-03-09T00:25:08.284 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6459999999999999e-06 2020-03-09T00:25:08.284 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.285 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844879+0000", 2020-03-09T00:25:08.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1310000000000002e-06 2020-03-09T00:25:08.285 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.285 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.285 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.286 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844883+0000", 2020-03-09T00:25:08.286 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4992e-05 2020-03-09T00:25:08.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844918+0000", 2020-03-09T00:25:08.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3540000000000001e-06 2020-03-09T00:25:08.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:08.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844919+0000", 2020-03-09T00:25:08.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97493126500000005 2020-03-09T00:25:08.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.819850+0000", 2020-03-09T00:25:08.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000196331 2020-03-09T00:25:08.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.820047+0000", 2020-03-09T00:25:08.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00092409100000000004 2020-03-09T00:25:08.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.291 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.291 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.820971+0000", 2020-03-09T00:25:08.291 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0026707079999999999 2020-03-09T00:25:08.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823641+0000", 2020-03-09T00:25:08.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017920241 2020-03-09T00:25:08.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.841562+0000", 2020-03-09T00:25:08.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9350000000000003e-05 2020-03-09T00:25:08.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.841651+0000", 2020-03-09T00:25:08.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99754793900000005 2020-03-09T00:25:08.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.294 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.294 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:405 3.1s0 3:81f9c489:::smithi10428818-29:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.844912+0000", 2020-03-09T00:25:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3513469029999996, 2020-03-09T00:25:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99692038400000005, 2020-03-09T00:25:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.295 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 405 2020-03-09T00:25:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844912+0000", 2020-03-09T00:25:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.297 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844912+0000", 2020-03-09T00:25:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9397000000000002e-05 2020-03-09T00:25:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844952+0000", 2020-03-09T00:25:08.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.267e-06 2020-03-09T00:25:08.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844956+0000", 2020-03-09T00:25:08.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1510000000000001e-06 2020-03-09T00:25:08.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844962+0000", 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.689e-06 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844965+0000", 2020-03-09T00:25:08.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6394999999999999e-05 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845011+0000", 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.114e-06 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:08.302 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845013+0000", 2020-03-09T00:25:08.302 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97726500100000002 2020-03-09T00:25:08.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.822278+0000", 2020-03-09T00:25:08.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.5245000000000002e-05 2020-03-09T00:25:08.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.822354+0000", 2020-03-09T00:25:08.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00072330500000000004 2020-03-09T00:25:08.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823077+0000", 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00072170800000000005 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823799+0000", 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.01796302 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.306 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.841762+0000", 2020-03-09T00:25:08.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1092000000000003e-05 2020-03-09T00:25:08.306 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.306 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.306 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.841833+0000", 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99692038400000005 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:406 3.1s0 3:81f9c489:::smithi10428818-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:08.307 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.844991+0000", 2020-03-09T00:25:08.308 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3512683699999997, 2020-03-09T00:25:08.308 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99952336500000005, 2020-03-09T00:25:08.308 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.308 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:08.308 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.308 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.308 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.309 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 406 2020-03-09T00:25:08.309 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.309 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.309 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.309 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.309 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844991+0000", 2020-03-09T00:25:08.309 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.310 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.310 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.310 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.844991+0000", 2020-03-09T00:25:08.310 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3319000000000002e-05 2020-03-09T00:25:08.310 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.311 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.311 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845014+0000", 2020-03-09T00:25:08.311 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9809999999999999e-06 2020-03-09T00:25:08.311 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.311 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845017+0000", 2020-03-09T00:25:08.312 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0350000000000004e-06 2020-03-09T00:25:08.312 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.312 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.312 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.312 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845024+0000", 2020-03-09T00:25:08.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4169999999999999e-06 2020-03-09T00:25:08.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.313 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.313 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.313 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845027+0000", 2020-03-09T00:25:08.313 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.0720999999999999e-05 2020-03-09T00:25:08.313 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.314 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.314 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845057+0000", 2020-03-09T00:25:08.314 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.091e-06 2020-03-09T00:25:08.314 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.314 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845058+0000", 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97822827000000001 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823287+0000", 2020-03-09T00:25:08.315 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.0505000000000003e-05 2020-03-09T00:25:08.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.316 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:08.316 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823357+0000", 2020-03-09T00:25:08.316 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018527077999999999 2020-03-09T00:25:08.316 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.316 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.317 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.317 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.841884+0000", 2020-03-09T00:25:08.317 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111599 2020-03-09T00:25:08.317 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.318 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.318 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.318 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.841996+0000", 2020-03-09T00:25:08.318 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002518349 2020-03-09T00:25:08.319 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.319 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.319 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.319 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.844514+0000", 2020-03-09T00:25:08.319 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99952336500000005 2020-03-09T00:25:08.320 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.320 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.320 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.320 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.320 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.320 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:407 3.1s0 3:85442c74:::smithi10428818-19:head [copy-from ver 4] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.320 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.845053+0000", 2020-03-09T00:25:08.321 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3512058659999999, 2020-03-09T00:25:08.321 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99790065800000005, 2020-03-09T00:25:08.321 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.321 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.321 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.321 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.321 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.322 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 407 2020-03-09T00:25:08.322 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.322 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.322 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.322 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.322 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845053+0000", 2020-03-09T00:25:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845053+0000", 2020-03-09T00:25:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6047999999999998e-05 2020-03-09T00:25:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845089+0000", 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.168e-06 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.324 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845093+0000", 2020-03-09T00:25:08.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9240000000000004e-06 2020-03-09T00:25:08.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.325 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.325 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.325 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845098+0000", 2020-03-09T00:25:08.325 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1349999999999999e-06 2020-03-09T00:25:08.325 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.326 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845101+0000", 2020-03-09T00:25:08.326 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9199999999999997e-05 2020-03-09T00:25:08.326 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.326 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.326 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845140+0000", 2020-03-09T00:25:08.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.683e-06 2020-03-09T00:25:08.327 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.327 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.327 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:08.327 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845142+0000", 2020-03-09T00:25:08.327 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97825159399999995 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823393+0000", 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000116815 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.328 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823510+0000", 2020-03-09T00:25:08.329 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086934109999999998 2020-03-09T00:25:08.329 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.329 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.329 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.329 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.832203+0000", 2020-03-09T00:25:08.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0044452500000000004 2020-03-09T00:25:08.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.330 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.330 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.330 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.836649+0000", 2020-03-09T00:25:08.330 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0062444079999999999 2020-03-09T00:25:08.330 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.331 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.331 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:08.331 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.842893+0000", 2020-03-09T00:25:08.331 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1022e-05 2020-03-09T00:25:08.331 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.331 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.331 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.332 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.842954+0000", 2020-03-09T00:25:08.332 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99790065800000005 2020-03-09T00:25:08.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.332 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.332 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.332 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.332 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.333 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:408 3.1s0 3:85442c74:::smithi10428818-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:08.333 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.845125+0000", 2020-03-09T00:25:08.333 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 4.3511343729999998, 2020-03-09T00:25:08.334 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99882568000000005, 2020-03-09T00:25:08.334 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.335 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:08.335 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:08.335 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:08.335 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.335 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 408 2020-03-09T00:25:08.335 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.335 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845125+0000", 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.336 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845125+0000", 2020-03-09T00:25:08.337 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3836e-05 2020-03-09T00:25:08.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.337 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.337 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.337 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845149+0000", 2020-03-09T00:25:08.337 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.965e-06 2020-03-09T00:25:08.337 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.338 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845152+0000", 2020-03-09T00:25:08.338 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9689999999999999e-06 2020-03-09T00:25:08.338 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.338 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.338 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845158+0000", 2020-03-09T00:25:08.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2900000000000001e-06 2020-03-09T00:25:08.339 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.339 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.339 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.339 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845160+0000", 2020-03-09T00:25:08.339 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2753999999999998e-05 2020-03-09T00:25:08.340 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.340 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.340 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.341 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845193+0000", 2020-03-09T00:25:08.341 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.175e-06 2020-03-09T00:25:08.341 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.341 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.341 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:08.341 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.845194+0000", 2020-03-09T00:25:08.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97847798399999997 2020-03-09T00:25:08.342 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.342 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.342 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.342 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823672+0000", 2020-03-09T00:25:08.342 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9762000000000002e-05 2020-03-09T00:25:08.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.343 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.343 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:08.343 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.823722+0000", 2020-03-09T00:25:08.343 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011527989000000001 2020-03-09T00:25:08.343 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.835250+0000", 2020-03-09T00:25:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6403999999999996e-05 2020-03-09T00:25:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:08.345 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.835316+0000", 2020-03-09T00:25:08.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083953280000000005 2020-03-09T00:25:08.345 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.345 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.345 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.345 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.843711+0000", 2020-03-09T00:25:08.345 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000111167 2020-03-09T00:25:08.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.346 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.346 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:08.346 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.843822+0000", 2020-03-09T00:25:08.346 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000128057 2020-03-09T00:25:08.346 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.347 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.347 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.347 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.843951+0000", 2020-03-09T00:25:08.347 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99882568000000005 2020-03-09T00:25:08.347 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.347 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.347 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 55/54 ECSubWriteReply(tid=62, last_complete=55'26, committed=1, applied=1))", 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.812393+0000", 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.3838660360000001, 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012629089, 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.348 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.349 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.349 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.349 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.349 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812393+0000", 2020-03-09T00:25:08.349 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.349 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812393+0000", 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3367000000000005e-05 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.350 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812457+0000", 2020-03-09T00:25:08.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9039999999999998e-06 2020-03-09T00:25:08.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.351 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.351 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.351 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812465+0000", 2020-03-09T00:25:08.351 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6228000000000001e-05 2020-03-09T00:25:08.351 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812482+0000", 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8489999999999995e-06 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.352 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812491+0000", 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012515379 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.825006+0000", 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6362e-05 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.825022+0000", 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012629089 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.354 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.355 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 55/54 ECSubWriteReply(tid=62, last_complete=55'26, committed=1, applied=1))", 2020-03-09T00:25:08.355 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.813288+0000", 2020-03-09T00:25:08.355 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.382971119, 2020-03-09T00:25:08.355 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011836238000000001, 2020-03-09T00:25:08.355 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.355 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.355 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813288+0000", 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.356 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813288+0000", 2020-03-09T00:25:08.357 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8046999999999994e-05 2020-03-09T00:25:08.357 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.357 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.357 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.357 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813356+0000", 2020-03-09T00:25:08.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.8900000000000002e-06 2020-03-09T00:25:08.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.358 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.358 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.358 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813366+0000", 2020-03-09T00:25:08.358 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6107e-05 2020-03-09T00:25:08.358 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.359 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813382+0000", 2020-03-09T00:25:08.359 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2539999999999998e-06 2020-03-09T00:25:08.359 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.359 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.359 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813388+0000", 2020-03-09T00:25:08.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011650445000000001 2020-03-09T00:25:08.360 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.360 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.360 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.360 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.825039+0000", 2020-03-09T00:25:08.360 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5494999999999994e-05 2020-03-09T00:25:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.825124+0000", 2020-03-09T00:25:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011836238000000001 2020-03-09T00:25:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.361 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 55/54 ECSubWriteReply(tid=62, last_complete=55'26, committed=1, applied=1))", 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.816229+0000", 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 3.380030171, 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090356900000000007, 2020-03-09T00:25:08.362 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.363 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.363 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.363 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.363 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.363 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816229+0000", 2020-03-09T00:25:08.363 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.363 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816229+0000", 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1908000000000004e-05 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.364 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816291+0000", 2020-03-09T00:25:08.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0099999999999995e-06 2020-03-09T00:25:08.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.365 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.365 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.365 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816299+0000", 2020-03-09T00:25:08.365 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5262000000000001e-05 2020-03-09T00:25:08.365 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.366 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816314+0000", 2020-03-09T00:25:08.366 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.69e-06 2020-03-09T00:25:08.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.366 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.367 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816319+0000", 2020-03-09T00:25:08.367 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0089364590000000008 2020-03-09T00:25:08.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.369 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.369 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.825255+0000", 2020-03-09T00:25:08.369 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.3610000000000007e-06 2020-03-09T00:25:08.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.370 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.370 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.370 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.825265+0000", 2020-03-09T00:25:08.370 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090356900000000007 2020-03-09T00:25:08.370 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.370 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.370 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.371 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.371 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s6 58/32 ECSubWrite(tid=149, reqid=osd.12.0:149, at_version=58'69, trim_to=0'0, roll_forward_to=57'67))", 2020-03-09T00:25:08.371 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:07.806694+0000", 2020-03-09T00:25:08.371 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.38956489999999999, 2020-03-09T00:25:08.371 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014682268, 2020-03-09T00:25:08.371 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.372 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.372 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.372 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.372 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.372 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.806694+0000", 2020-03-09T00:25:08.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.373 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.806694+0000", 2020-03-09T00:25:08.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8875999999999997e-05 2020-03-09T00:25:08.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.373 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.806753+0000", 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5440999999999999e-05 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.806779+0000", 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4246999999999999e-05 2020-03-09T00:25:08.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.806793+0000", 2020-03-09T00:25:08.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1600000000000003e-06 2020-03-09T00:25:08.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.806799+0000", 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000588848 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.807388+0000", 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4899000000000001e-05 2020-03-09T00:25:08.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.377 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.377 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.807403+0000", 2020-03-09T00:25:08.377 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013918866 2020-03-09T00:25:08.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.377 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.378 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.821322+0000", 2020-03-09T00:25:08.378 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4931e-05 2020-03-09T00:25:08.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.378 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.378 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.821377+0000", 2020-03-09T00:25:08.378 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014682268 2020-03-09T00:25:08.379 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.379 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.379 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.379 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.379 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.379 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s6 58/32 ECSubWrite(tid=151, reqid=osd.12.0:151, at_version=58'71, trim_to=0'0, roll_forward_to=57'67))", 2020-03-09T00:25:08.379 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:07.808972+0000", 2020-03-09T00:25:08.380 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.38728688700000002, 2020-03-09T00:25:08.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017981543999999999, 2020-03-09T00:25:08.380 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.380 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.380 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.380 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.808972+0000", 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.808972+0000", 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4871999999999998e-05 2020-03-09T00:25:08.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.382 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.382 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809037+0000", 2020-03-09T00:25:08.382 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5335000000000001e-05 2020-03-09T00:25:08.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.382 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809063+0000", 2020-03-09T00:25:08.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5605000000000001e-05 2020-03-09T00:25:08.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809078+0000", 2020-03-09T00:25:08.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4739999999999999e-06 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809086+0000", 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001691098 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.384 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.810777+0000", 2020-03-09T00:25:08.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3978000000000001e-05 2020-03-09T00:25:08.385 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.385 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.385 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.385 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.810791+0000", 2020-03-09T00:25:08.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016115072000000001 2020-03-09T00:25:08.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.386 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.826906+0000", 2020-03-09T00:25:08.386 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8109999999999998e-05 2020-03-09T00:25:08.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.826954+0000", 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017981543999999999 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.388 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 58/32 ECSubWrite(tid=141, reqid=osd.11.0:141, at_version=58'34, trim_to=0'0, roll_forward_to=57'32))", 2020-03-09T00:25:08.388 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:07.812840+0000", 2020-03-09T00:25:08.388 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.38341936500000001, 2020-03-09T00:25:08.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011401700000000001, 2020-03-09T00:25:08.388 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.388 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.388 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.812840+0000", 2020-03-09T00:25:08.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.812840+0000", 2020-03-09T00:25:08.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8751999999999995e-05 2020-03-09T00:25:08.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.390 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.812909+0000", 2020-03-09T00:25:08.391 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6520999999999999e-05 2020-03-09T00:25:08.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.812935+0000", 2020-03-09T00:25:08.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5387999999999999e-05 2020-03-09T00:25:08.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.394 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.812951+0000", 2020-03-09T00:25:08.394 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8679999999999999e-06 2020-03-09T00:25:08.394 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.394 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.394 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.395 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.812958+0000", 2020-03-09T00:25:08.395 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5856999999999997e-05 2020-03-09T00:25:08.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.395 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.395 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813014+0000", 2020-03-09T00:25:08.395 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2582000000000001e-05 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813027+0000", 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011161096000000001 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.396 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824188+0000", 2020-03-09T00:25:08.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3635999999999999e-05 2020-03-09T00:25:08.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824242+0000", 2020-03-09T00:25:08.397 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011401700000000001 2020-03-09T00:25:08.398 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.398 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.398 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.398 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs6 58/32 ECSubWrite(tid=143, reqid=osd.11.0:143, at_version=58'36, trim_to=0'0, roll_forward_to=57'32))", 2020-03-09T00:25:08.398 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:07.813013+0000", 2020-03-09T00:25:08.399 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.38324595, 2020-03-09T00:25:08.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011688551, 2020-03-09T00:25:08.399 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.399 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.399 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813013+0000", 2020-03-09T00:25:08.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813013+0000", 2020-03-09T00:25:08.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4209999999999998e-05 2020-03-09T00:25:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813067+0000", 2020-03-09T00:25:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6767999999999999e-05 2020-03-09T00:25:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813094+0000", 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3852000000000001e-05 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813108+0000", 2020-03-09T00:25:08.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2940000000000002e-06 2020-03-09T00:25:08.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813114+0000", 2020-03-09T00:25:08.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067902700000000002 2020-03-09T00:25:08.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.819905+0000", 2020-03-09T00:25:08.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2422999999999999e-05 2020-03-09T00:25:08.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.819917+0000", 2020-03-09T00:25:08.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0047324369999999999 2020-03-09T00:25:08.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824650+0000", 2020-03-09T00:25:08.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2296999999999998e-05 2020-03-09T00:25:08.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824702+0000", 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011688551 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.407 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s5 58/54 ECSubWrite(tid=23, reqid=osd.0.0:23, at_version=58'32, trim_to=0'0, roll_forward_to=57'28))", 2020-03-09T00:25:08.407 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:07.814945+0000", 2020-03-09T00:25:08.407 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.38131420599999999, 2020-03-09T00:25:08.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0091276359999999997, 2020-03-09T00:25:08.407 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:08.407 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.814945+0000", 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:08.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.814945+0000", 2020-03-09T00:25:08.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8081999999999997e-05 2020-03-09T00:25:08.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.409 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:08.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.815003+0000", 2020-03-09T00:25:08.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2064999999999999e-05 2020-03-09T00:25:08.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.410 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:08.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.815035+0000", 2020-03-09T00:25:08.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.632e-05 2020-03-09T00:25:08.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.815051+0000", 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.554e-06 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.815059+0000", 2020-03-09T00:25:08.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001453277 2020-03-09T00:25:08.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.412 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:08.412 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.816512+0000", 2020-03-09T00:25:08.412 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.238e-05 2020-03-09T00:25:08.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.816525+0000", 2020-03-09T00:25:08.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074966190000000004 2020-03-09T00:25:08.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824021+0000", 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1338999999999998e-05 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824073+0000", 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0091276359999999997 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.414 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.415 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.415 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:08.415 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:08.416 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:08.417 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:08.416+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:08.418 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:08.518 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:08.517+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:08.555 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:08.555 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:08.555 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:08.564 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:08.622 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:08.618+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:08.689 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:08.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:08.689 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:08.690 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:08.690 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:08.698 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:08.720 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:08.719+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:08.783 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-03-09T00:25:08.812 INFO:tasks.rados.rados.0.smithi104.stdout:187: done (2 left) 2020-03-09T00:25:08.812 INFO:tasks.rados.rados.0.smithi104.stdout:188: rollback oid 31 current snap is 14 2020-03-09T00:25:08.813 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 31 to 12 2020-03-09T00:25:08.813 INFO:tasks.rados.rados.0.smithi104.stdout:185: finishing write tid 1 to smithi10428818-50 2020-03-09T00:25:08.813 INFO:tasks.rados.rados.0.smithi104.stdout:185: finishing write tid 2 to smithi10428818-50 2020-03-09T00:25:08.813 INFO:tasks.rados.rados.0.smithi104.stdout:185: finishing write tid 3 to smithi10428818-50 2020-03-09T00:25:08.813 INFO:tasks.rados.rados.0.smithi104.stdout:185: finishing write tid 5 to smithi10428818-50 2020-03-09T00:25:08.813 INFO:tasks.rados.rados.0.smithi104.stdout:185: finishing write tid 6 to smithi10428818-50 2020-03-09T00:25:08.813 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 50 v 28 (ObjNum 91 snap 14 seq_num 91) dirty exists 2020-03-09T00:25:08.814 INFO:tasks.rados.rados.0.smithi104.stdout:185: left oid 50 (ObjNum 91 snap 14 seq_num 91) 2020-03-09T00:25:08.814 INFO:tasks.rados.rados.0.smithi104.stdout:183: finishing copy_from to smithi10428818-34 2020-03-09T00:25:08.814 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 34 v 35 (ObjNum 71 snap 11 seq_num 71) dirty exists 2020-03-09T00:25:08.814 INFO:tasks.rados.rados.0.smithi104.stdout:183: finishing copy_from racing read to smithi10428818-34 2020-03-09T00:25:08.814 INFO:tasks.rados.rados.0.smithi104.stdout:183: done (2 left) 2020-03-09T00:25:08.815 INFO:tasks.rados.rados.0.smithi104.stdout:185: done (1 left) 2020-03-09T00:25:08.815 INFO:tasks.rados.rados.0.smithi104.stdout:189: copy_from oid 7 from oid 5 current snap is 14 2020-03-09T00:25:08.815 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 3 current snap is 14 2020-03-09T00:25:08.815 INFO:tasks.rados.rados.0.smithi104.stdout:190: seq_num 92 ranges {49152=786432,835584=491520,1327104=278528} 2020-03-09T00:25:08.817 INFO:tasks.rados.rados.0.smithi104.stdout:190: writing smithi10428818-3 from 49152 to 835584 tid 1 2020-03-09T00:25:08.821 INFO:tasks.rados.rados.0.smithi104.stdout:190: writing smithi10428818-3 from 835584 to 1327104 tid 2 2020-03-09T00:25:08.821 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:08.819+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:08.824 INFO:tasks.rados.rados.0.smithi104.stdout:190: writing smithi10428818-3 from 1327104 to 1605632 tid 3 2020-03-09T00:25:08.824 INFO:tasks.rados.rados.0.smithi104.stdout:191: read oid 29 snap -1 2020-03-09T00:25:08.824 INFO:tasks.rados.rados.0.smithi104.stdout:191: expect (ObjNum 80 snap 12 seq_num 80) 2020-03-09T00:25:08.825 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 19 current snap is 14 2020-03-09T00:25:08.825 INFO:tasks.rados.rados.0.smithi104.stdout:192: seq_num 93 ranges {229376=802816,1032192=638976,1671168=131072} 2020-03-09T00:25:08.829 INFO:tasks.rados.rados.0.smithi104.stdout:192: writing smithi10428818-19 from 229376 to 1032192 tid 1 2020-03-09T00:25:08.834 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:08.834 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs3 39/32 ECSubWrite(tid=22, reqid=client.4751.0:210, at_version=39'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:25:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:48.828466+0000", 2020-03-09T00:25:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.002396349000001, 2020-03-09T00:25:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004661971, 2020-03-09T00:25:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.838 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.828466+0000", 2020-03-09T00:25:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.828466+0000", 2020-03-09T00:25:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067655999999999999 2020-03-09T00:25:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829142+0000", 2020-03-09T00:25:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6149e-05 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829169+0000", 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6626e-05 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829185+0000", 2020-03-09T00:25:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2470000000000002e-06 2020-03-09T00:25:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829192+0000", 2020-03-09T00:25:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2005e-05 2020-03-09T00:25:08.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829254+0000", 2020-03-09T00:25:08.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5999e-05 2020-03-09T00:25:08.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.829270+0000", 2020-03-09T00:25:08.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038224880000000002 2020-03-09T00:25:08.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.833093+0000", 2020-03-09T00:25:08.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4897000000000001e-05 2020-03-09T00:25:08.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:48.833128+0000", 2020-03-09T00:25:08.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004661971 2020-03-09T00:25:08.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.847 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.847 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.847 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:08.847 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:08.847 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.976739286999999, 2020-03-09T00:25:08.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010037961, 2020-03-09T00:25:08.848 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.848 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.848 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:08.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:08.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040949200000000002 2020-03-09T00:25:08.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854532+0000", 2020-03-09T00:25:08.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1631000000000003e-05 2020-03-09T00:25:08.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854564+0000", 2020-03-09T00:25:08.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2872000000000001e-05 2020-03-09T00:25:08.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854587+0000", 2020-03-09T00:25:08.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0699999999999999e-05 2020-03-09T00:25:08.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854598+0000", 2020-03-09T00:25:08.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6437000000000001e-05 2020-03-09T00:25:08.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854664+0000", 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6180999999999998e-05 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854700+0000", 2020-03-09T00:25:08.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094130399999999993 2020-03-09T00:25:08.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864113+0000", 2020-03-09T00:25:08.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7608000000000002e-05 2020-03-09T00:25:08.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864161+0000", 2020-03-09T00:25:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010037961 2020-03-09T00:25:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.857 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:08.857 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.856927+0000", 2020-03-09T00:25:08.857 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.973934723999999, 2020-03-09T00:25:08.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072822809999999998, 2020-03-09T00:25:08.857 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.858 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.858 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856927+0000", 2020-03-09T00:25:08.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856927+0000", 2020-03-09T00:25:08.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014181799999999999 2020-03-09T00:25:08.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857069+0000", 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7234000000000001e-05 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857087+0000", 2020-03-09T00:25:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7180000000000008e-06 2020-03-09T00:25:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857096+0000", 2020-03-09T00:25:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8880000000000003e-06 2020-03-09T00:25:08.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857105+0000", 2020-03-09T00:25:08.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040913800000000004 2020-03-09T00:25:08.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.861197+0000", 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3682e-05 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.861210+0000", 2020-03-09T00:25:08.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002965152 2020-03-09T00:25:08.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864175+0000", 2020-03-09T00:25:08.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4409000000000002e-05 2020-03-09T00:25:08.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864210+0000", 2020-03-09T00:25:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072822809999999998 2020-03-09T00:25:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs3 44/32 ECSubWrite(tid=34, reqid=client.4751.0:249, at_version=44'16, trim_to=0'0, roll_forward_to=44'14))", 2020-03-09T00:25:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.867459+0000", 2020-03-09T00:25:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.963403614000001, 2020-03-09T00:25:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046205660000000004, 2020-03-09T00:25:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.866 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.867 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867459+0000", 2020-03-09T00:25:08.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867459+0000", 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030457500000000002 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867763+0000", 2020-03-09T00:25:08.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0179999999999999e-05 2020-03-09T00:25:08.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867793+0000", 2020-03-09T00:25:08.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3425e-05 2020-03-09T00:25:08.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867817+0000", 2020-03-09T00:25:08.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3130000000000004e-06 2020-03-09T00:25:08.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867826+0000", 2020-03-09T00:25:08.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2817999999999994e-05 2020-03-09T00:25:08.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867889+0000", 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6288999999999999e-05 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867905+0000", 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041335709999999999 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.872039+0000", 2020-03-09T00:25:08.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0395000000000002e-05 2020-03-09T00:25:08.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.872079+0000", 2020-03-09T00:25:08.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046205660000000004 2020-03-09T00:25:08.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.874 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.875 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:08.875 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.817491+0000", 2020-03-09T00:25:08.875 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.013371328, 2020-03-09T00:25:08.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056599119999999996, 2020-03-09T00:25:08.876 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.876 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.876 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817491+0000", 2020-03-09T00:25:08.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817491+0000", 2020-03-09T00:25:08.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032379400000000002 2020-03-09T00:25:08.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817815+0000", 2020-03-09T00:25:08.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3218999999999999e-05 2020-03-09T00:25:08.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817838+0000", 2020-03-09T00:25:08.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2707999999999999e-05 2020-03-09T00:25:08.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817851+0000", 2020-03-09T00:25:08.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3379999999999998e-06 2020-03-09T00:25:08.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817855+0000", 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0161999999999997e-05 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817925+0000", 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4869999999999996e-06 2020-03-09T00:25:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817933+0000", 2020-03-09T00:25:08.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051815780000000001 2020-03-09T00:25:08.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823114+0000", 2020-03-09T00:25:08.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6625999999999998e-05 2020-03-09T00:25:08.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823151+0000", 2020-03-09T00:25:08.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056599119999999996 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:296 3.2s0 3:49883c8e:::smithi10428818-40:head [create,setxattr eROReghfPdlf^m`e`^hhNZXOkdsQ]]kX`senUiSZ (875) in=40b,setxattr s`rNbaqcP\\diTaoZYXg[m[inZ`]QtZdjom`_dhUX (1928) in=40b,setxattr [[YXf (69) in=5b,setxattr ]rjrtd]YkcY\\lQ[bOp_lNWs\\NOirOQ[Ri`rndifj (5260) in=40b,setxattr n[ZSX[Rc`\\dlYpV^rPdsQfnbg^l]ioSo^`oZsXaf (19016) in=40b,setxattr b\\ffPRnhZeR\\shWQYPRaZVroNXYoUYf[eemPeQen (1648) in=40b,setxattr [gNas\\_VhjX_YRX[aQbekgbWYg`P_SWgskeiZbkP (1333) in=40b,setxattr Ysi^k[[WWsYagaeecamn^bcecbfS]p^i^jfgsmTY (6736) in=40b,setxattr Nm_N]c]XYfmpWd\\aiW^bPSdY]TjNrnirngZdd_jb (103) in=40b,setxattr cisVjO^iia\\[tQaXjaroW`VRNl[aiYR\\tcnhVsfb (8841) in=40b,setxattr Zgg]\\aOQfmaVkY (69) in=14b,setxattr X`i_`tsrZ[XRfqacNap`[O`nSf]O`[Z[Oc[aenTc (1647) in=40b,setxattr oV[OofncW[WdO[XrprmdpkoptWZRfcPm^Rkernnf (34) in=40b,setxattr \\UQYgqaahhcRrkP`h`SgamahYafPmsojP_snXkmc (314) in=40b,setxattr pjilgXb]Qkl]gt__a_ZrtRptOPONOVrsmqcZ__^T (840) in=40b,setxattr Wa`ZaT[b]kbqdZrfSUkVafjqXod\\qd (69) in=30b,setxattr [YaXgtehetrqtspbYYY[ZqritVsbZeW[ak`ma``l (418) in=40b,setxattr hPanNtTRrfi`tYcdO]eRl]`fj`hllRNnXThl\\YO (69) in=39b,setxattr S_`Ri[Qj]fZqXkqqZYcte[N]Wq\\cQ]PlQdllcb]s (1542) in=40b,setxattr ttkZc`XVXmWiqdYWeagrmr^i[OUs^jpafRbOUWdp (594) in=40b] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.822533+0000", 2020-03-09T00:25:08.884 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.008329419000001, 2020-03-09T00:25:08.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051422450000000001, 2020-03-09T00:25:08.885 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.885 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:08.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:08.885 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.885 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 296 2020-03-09T00:25:08.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.886 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822533+0000", 2020-03-09T00:25:08.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822533+0000", 2020-03-09T00:25:08.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112026 2020-03-09T00:25:08.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822645+0000", 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9469999999999998e-06 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822655+0000", 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.046e-05 2020-03-09T00:25:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822665+0000", 2020-03-09T00:25:08.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8730000000000004e-06 2020-03-09T00:25:08.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822671+0000", 2020-03-09T00:25:08.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9454999999999999e-05 2020-03-09T00:25:08.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822731+0000", 2020-03-09T00:25:08.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022593400000000001 2020-03-09T00:25:08.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:08.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822956+0000", 2020-03-09T00:25:08.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00094100800000000001 2020-03-09T00:25:08.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823897+0000", 2020-03-09T00:25:08.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011360719999999999 2020-03-09T00:25:08.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.825034+0000", 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025793629999999999 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827613+0000", 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2106999999999993e-05 2020-03-09T00:25:08.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827675+0000", 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051422450000000001 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.895 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 49/48 ECSubWrite(tid=76, reqid=client.4751.0:299, at_version=49'30, trim_to=0'0, roll_forward_to=49'29))", 2020-03-09T00:25:08.895 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.826879+0000", 2020-03-09T00:25:08.895 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.003983034999999, 2020-03-09T00:25:08.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047255420000000001, 2020-03-09T00:25:08.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.895 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.826879+0000", 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.826879+0000", 2020-03-09T00:25:08.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015170900000000001 2020-03-09T00:25:08.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827031+0000", 2020-03-09T00:25:08.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4344e-05 2020-03-09T00:25:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827045+0000", 2020-03-09T00:25:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2399999999999996e-06 2020-03-09T00:25:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827054+0000", 2020-03-09T00:25:08.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3109999999999999e-06 2020-03-09T00:25:08.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827059+0000", 2020-03-09T00:25:08.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3791999999999997e-05 2020-03-09T00:25:08.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827103+0000", 2020-03-09T00:25:08.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2359e-05 2020-03-09T00:25:08.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.827115+0000", 2020-03-09T00:25:08.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044456039999999997 2020-03-09T00:25:08.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.831561+0000", 2020-03-09T00:25:08.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4183e-05 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.831605+0000", 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047255420000000001 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.904 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:08.904 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:08.904 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.9795235169999996, 2020-03-09T00:25:08.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649, 2020-03-09T00:25:08.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:08.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022148699999999999 2020-03-09T00:25:08.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853554+0000", 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7487e-05 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853571+0000", 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1065e-05 2020-03-09T00:25:08.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853582+0000", 2020-03-09T00:25:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7259999999999999e-06 2020-03-09T00:25:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853588+0000", 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6578e-05 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853634+0000", 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1282e-05 2020-03-09T00:25:08.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853646+0000", 2020-03-09T00:25:08.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012718002000000001 2020-03-09T00:25:08.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866364+0000", 2020-03-09T00:25:08.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6638999999999999e-05 2020-03-09T00:25:08.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866400+0000", 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.913 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:375 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.913 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:08.913 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0300755029999999, 2020-03-09T00:25:08.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317, 2020-03-09T00:25:08.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.913 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.913 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:08.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:08.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.914 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 375 2020-03-09T00:25:08.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.914 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:08.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:08.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001381363 2020-03-09T00:25:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802168+0000", 2020-03-09T00:25:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5639000000000001e-05 2020-03-09T00:25:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802184+0000", 2020-03-09T00:25:08.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6752000000000001e-05 2020-03-09T00:25:08.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802240+0000", 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7064999999999999e-05 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802268+0000", 2020-03-09T00:25:08.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6162999999999996e-05 2020-03-09T00:25:08.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802344+0000", 2020-03-09T00:25:08.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000166458 2020-03-09T00:25:08.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:08.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802510+0000", 2020-03-09T00:25:08.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018314410000000001 2020-03-09T00:25:08.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804342+0000", 2020-03-09T00:25:08.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079462270000000002 2020-03-09T00:25:08.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812288+0000", 2020-03-09T00:25:08.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078060580000000003 2020-03-09T00:25:08.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:08.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820094+0000", 2020-03-09T00:25:08.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110151 2020-03-09T00:25:08.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820204+0000", 2020-03-09T00:25:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317 2020-03-09T00:25:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.924 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:376 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.924 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:08.925 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0267201679999998, 2020-03-09T00:25:08.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127, 2020-03-09T00:25:08.925 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.925 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.925 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:08.925 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:08.926 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.926 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 376 2020-03-09T00:25:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00133916 2020-03-09T00:25:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805481+0000", 2020-03-09T00:25:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1296e-05 2020-03-09T00:25:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805492+0000", 2020-03-09T00:25:08.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1435999999999999e-05 2020-03-09T00:25:08.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805514+0000", 2020-03-09T00:25:08.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5969999999999998e-06 2020-03-09T00:25:08.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805524+0000", 2020-03-09T00:25:08.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026245270000000002 2020-03-09T00:25:08.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808148+0000", 2020-03-09T00:25:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000157388 2020-03-09T00:25:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808305+0000", 2020-03-09T00:25:08.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00107099 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.809376+0000", 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071133840000000004 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816490+0000", 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038222479999999999 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820312+0000", 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012910100000000001 2020-03-09T00:25:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820441+0000", 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:377 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~245760] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.022860756, 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858, 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.936 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 377 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:08.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:08.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00047463399999999998 2020-03-09T00:25:08.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808476+0000", 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7579999999999998e-06 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808482+0000", 2020-03-09T00:25:08.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7729999999999994e-06 2020-03-09T00:25:08.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808492+0000", 2020-03-09T00:25:08.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:08.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808496+0000", 2020-03-09T00:25:08.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003930813 2020-03-09T00:25:08.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812427+0000", 2020-03-09T00:25:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013734399999999999 2020-03-09T00:25:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812564+0000", 2020-03-09T00:25:08.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00094040499999999995 2020-03-09T00:25:08.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.813505+0000", 2020-03-09T00:25:08.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030313029999999999 2020-03-09T00:25:08.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816536+0000", 2020-03-09T00:25:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048022810000000003 2020-03-09T00:25:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821338+0000", 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4952999999999997e-05 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821433+0000", 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858 2020-03-09T00:25:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.946 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.946 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.946 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:378 3.2s0 3:482d6e06:::smithi10428818-17:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:08.947 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:08.947 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0223326320000004, 2020-03-09T00:25:08.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001, 2020-03-09T00:25:08.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.947 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:08.947 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:08.948 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:08.948 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.948 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 378 2020-03-09T00:25:08.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.948 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2327999999999999e-05 2020-03-09T00:25:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808572+0000", 2020-03-09T00:25:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7400000000000002e-06 2020-03-09T00:25:08.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808576+0000", 2020-03-09T00:25:08.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5849999999999996e-06 2020-03-09T00:25:08.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808582+0000", 2020-03-09T00:25:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7070000000000001e-06 2020-03-09T00:25:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808585+0000", 2020-03-09T00:25:08.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068015719999999997 2020-03-09T00:25:08.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815386+0000", 2020-03-09T00:25:08.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124704 2020-03-09T00:25:08.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:08.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815511+0000", 2020-03-09T00:25:08.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00063270399999999999 2020-03-09T00:25:08.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816144+0000", 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056543100000000003 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816709+0000", 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048141440000000002 2020-03-09T00:25:08.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821523+0000", 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012666000000000001 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821650+0000", 2020-03-09T00:25:08.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001 2020-03-09T00:25:08.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.957 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.957 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:379 3.2s0 3:482d6e06:::smithi10428818-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:08.958 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:08.958 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0222526119999999, 2020-03-09T00:25:08.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999, 2020-03-09T00:25:08.958 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.958 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:08.958 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:08.958 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 379 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:08.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:08.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3624000000000002e-05 2020-03-09T00:25:08.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808643+0000", 2020-03-09T00:25:08.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.625e-06 2020-03-09T00:25:08.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808647+0000", 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1500000000000002e-06 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808654+0000", 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8480000000000001e-06 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808657+0000", 2020-03-09T00:25:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079120670000000001 2020-03-09T00:25:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816569+0000", 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012729000000000001 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816696+0000", 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054928110000000002 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822189+0000", 2020-03-09T00:25:08.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6662000000000005e-05 2020-03-09T00:25:08.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:08.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822276+0000", 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001870029 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.824146+0000", 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.966 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.967 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 54/48 ECSubWrite(tid=93, reqid=client.4751.0:380, at_version=54'42, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:08.967 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.813753+0000", 2020-03-09T00:25:08.967 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.017109638, 2020-03-09T00:25:08.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048002519999999996, 2020-03-09T00:25:08.968 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.968 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.968 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.813753+0000", 2020-03-09T00:25:08.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.813753+0000", 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073154600000000002 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.814484+0000", 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6514000000000001e-05 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.814511+0000", 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.022e-05 2020-03-09T00:25:08.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.814531+0000", 2020-03-09T00:25:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9949999999999999e-06 2020-03-09T00:25:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.814538+0000", 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1192999999999997e-05 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.814579+0000", 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5939999999999994e-06 2020-03-09T00:25:08.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.814588+0000", 2020-03-09T00:25:08.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039098759999999996 2020-03-09T00:25:08.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.818497+0000", 2020-03-09T00:25:08.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5313999999999999e-05 2020-03-09T00:25:08.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.818553+0000", 2020-03-09T00:25:08.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048002519999999996 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 54/48 ECSubWrite(tid=94, reqid=client.4751.0:381, at_version=54'43, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.819631+0000", 2020-03-09T00:25:08.975 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0112310689999999, 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047769659999999997, 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.819631+0000", 2020-03-09T00:25:08.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.819631+0000", 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072729999999999995 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820358+0000", 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6591000000000001e-05 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820375+0000", 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6223999999999999e-05 2020-03-09T00:25:08.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820391+0000", 2020-03-09T00:25:08.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2289999999999992e-06 2020-03-09T00:25:08.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820400+0000", 2020-03-09T00:25:08.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4048000000000002e-05 2020-03-09T00:25:08.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820455+0000", 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4017e-05 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820469+0000", 2020-03-09T00:25:08.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039036890000000001 2020-03-09T00:25:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.824372+0000", 2020-03-09T00:25:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5868000000000003e-05 2020-03-09T00:25:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.824408+0000", 2020-03-09T00:25:08.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047769659999999997 2020-03-09T00:25:08.984 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.984 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.984 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.985 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 54/32 ECSubWrite(tid=77, reqid=client.4751.0:415, at_version=54'44, trim_to=0'0, roll_forward_to=54'42))", 2020-03-09T00:25:08.985 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.202946+0000", 2020-03-09T00:25:08.985 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.6279158819999999, 2020-03-09T00:25:08.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046063090000000003, 2020-03-09T00:25:08.985 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.985 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.986 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202946+0000", 2020-03-09T00:25:08.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202946+0000", 2020-03-09T00:25:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000568535 2020-03-09T00:25:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203515+0000", 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3037999999999999e-05 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203538+0000", 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3302000000000001e-05 2020-03-09T00:25:08.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203551+0000", 2020-03-09T00:25:08.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3090000000000003e-06 2020-03-09T00:25:08.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203556+0000", 2020-03-09T00:25:08.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9380999999999999e-05 2020-03-09T00:25:08.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203616+0000", 2020-03-09T00:25:08.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2906000000000001e-05 2020-03-09T00:25:08.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:08.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203639+0000", 2020-03-09T00:25:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003862168 2020-03-09T00:25:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.207501+0000", 2020-03-09T00:25:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1669999999999998e-05 2020-03-09T00:25:08.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:08.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.207553+0000", 2020-03-09T00:25:08.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046063090000000003 2020-03-09T00:25:08.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.993 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:08.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:08.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.994 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s2 55/54 ECSubWrite(tid=62, reqid=client.4751.0:402, at_version=55'26, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:08.994 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.808002+0000", 2020-03-09T00:25:08.994 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0228605589999997, 2020-03-09T00:25:08.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067101260000000003, 2020-03-09T00:25:08.995 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:08.995 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:08.995 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:08.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:08.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808002+0000", 2020-03-09T00:25:08.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:08.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:08.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808002+0000", 2020-03-09T00:25:08.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022239199999999999 2020-03-09T00:25:08.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:08.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808224+0000", 2020-03-09T00:25:08.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7348999999999999e-05 2020-03-09T00:25:08.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:08.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808251+0000", 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1313000000000001e-05 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808273+0000", 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6549999999999993e-06 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808281+0000", 2020-03-09T00:25:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3673000000000001e-05 2020-03-09T00:25:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:08.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808315+0000", 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2488e-05 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808328+0000", 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063325869999999998 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.814660+0000", 2020-03-09T00:25:09.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1669000000000003e-05 2020-03-09T00:25:09.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:09.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.814712+0000", 2020-03-09T00:25:09.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067101260000000003 2020-03-09T00:25:09.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.002 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:09.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.002 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 55/32 ECSubWrite(tid=84, reqid=client.4751.0:433, at_version=55'49, trim_to=0'0, roll_forward_to=54'47))", 2020-03-09T00:25:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.810394+0000", 2020-03-09T00:25:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0204678009999997, 2020-03-09T00:25:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050081580000000004, 2020-03-09T00:25:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:09.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810394+0000", 2020-03-09T00:25:09.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:09.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:09.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810394+0000", 2020-03-09T00:25:09.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017586699999999999 2020-03-09T00:25:09.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:09.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810570+0000", 2020-03-09T00:25:09.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3981999999999998e-05 2020-03-09T00:25:09.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:09.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810594+0000", 2020-03-09T00:25:09.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9485999999999999e-05 2020-03-09T00:25:09.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:09.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810614+0000", 2020-03-09T00:25:09.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654000000000001e-05 2020-03-09T00:25:09.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810625+0000", 2020-03-09T00:25:09.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8931999999999998e-05 2020-03-09T00:25:09.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:09.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810684+0000", 2020-03-09T00:25:09.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3386e-05 2020-03-09T00:25:09.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.810698+0000", 2020-03-09T00:25:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046722259999999998 2020-03-09T00:25:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.815370+0000", 2020-03-09T00:25:09.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2625e-05 2020-03-09T00:25:09.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:09.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.815403+0000", 2020-03-09T00:25:09.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050081580000000004 2020-03-09T00:25:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s2 55/54 ECSubWrite(tid=67, reqid=client.4751.0:398, at_version=55'18, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.813412+0000", 2020-03-09T00:25:09.013 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0174504400000002, 2020-03-09T00:25:09.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066538960000000003, 2020-03-09T00:25:09.013 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:09.013 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:09.013 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:09.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:09.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813412+0000", 2020-03-09T00:25:09.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:09.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:09.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813412+0000", 2020-03-09T00:25:09.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023050499999999999 2020-03-09T00:25:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813642+0000", 2020-03-09T00:25:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6463000000000001e-05 2020-03-09T00:25:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:09.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813659+0000", 2020-03-09T00:25:09.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1501000000000001e-05 2020-03-09T00:25:09.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:09.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813670+0000", 2020-03-09T00:25:09.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5430000000000004e-06 2020-03-09T00:25:09.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813675+0000", 2020-03-09T00:25:09.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4376999999999999e-05 2020-03-09T00:25:09.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813719+0000", 2020-03-09T00:25:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2407e-05 2020-03-09T00:25:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813732+0000", 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006282953 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.820014+0000", 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1146999999999997e-05 2020-03-09T00:25:09.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:09.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.820066+0000", 2020-03-09T00:25:09.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066538960000000003 2020-03-09T00:25:09.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.020 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:09.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.021 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s2 55/54 ECSubWrite(tid=68, reqid=client.4751.0:399, at_version=55'19, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:09.021 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.813730+0000", 2020-03-09T00:25:09.021 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0171324979999996, 2020-03-09T00:25:09.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063762869999999996, 2020-03-09T00:25:09.021 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:09.022 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:09.022 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:09.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:09.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813730+0000", 2020-03-09T00:25:09.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813730+0000", 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020127 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:09.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813931+0000", 2020-03-09T00:25:09.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2733000000000001e-05 2020-03-09T00:25:09.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:09.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813954+0000", 2020-03-09T00:25:09.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7757999999999998e-05 2020-03-09T00:25:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813971+0000", 2020-03-09T00:25:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4490000000000002e-06 2020-03-09T00:25:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813979+0000", 2020-03-09T00:25:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020680490000000002 2020-03-09T00:25:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:09.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.816047+0000", 2020-03-09T00:25:09.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4752e-05 2020-03-09T00:25:09.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.816062+0000", 2020-03-09T00:25:09.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040079670000000003 2020-03-09T00:25:09.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.820070+0000", 2020-03-09T00:25:09.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6309000000000001e-05 2020-03-09T00:25:09.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:09.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:09.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:09.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.820106+0000", 2020-03-09T00:25:09.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063762869999999996 2020-03-09T00:25:09.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.029 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:09.030 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.030 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:09.030 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:09.030 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:09.030 INFO:tasks.rados.rados.0.smithi104.stdout:192: writing smithi10428818-19 from 1032192 to 1671168 tid 2 2020-03-09T00:25:09.030 INFO:tasks.rados.rados.0.smithi104.stdout:192: writing smithi10428818-19 from 1671168 to 1802240 tid 3 2020-03-09T00:25:09.030 INFO:tasks.rados.rados.0.smithi104.stdout:193: rollback oid 17 current snap is 14 2020-03-09T00:25:09.031 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 17 to 13 2020-03-09T00:25:09.031 INFO:tasks.rados.rados.0.smithi104.stdout:194: delete oid 38 current snap is 14 2020-03-09T00:25:09.033 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:25:09.034 INFO:tasks.rados.rados.0.smithi104.stdout:193: finishing rollback tid 0 to smithi10428818-17 2020-03-09T00:25:09.034 INFO:tasks.rados.rados.0.smithi104.stdout:193: finishing rollback tid 1 to smithi10428818-17 2020-03-09T00:25:09.034 INFO:tasks.rados.rados.0.smithi104.stdout:188: finishing rollback tid 0 to smithi10428818-31 2020-03-09T00:25:09.034 INFO:tasks.rados.rados.0.smithi104.stdout:188: finishing rollback tid 1 to smithi10428818-31 2020-03-09T00:25:09.034 INFO:tasks.rados.rados.0.smithi104.stdout:191: expect (ObjNum 80 snap 12 seq_num 80) 2020-03-09T00:25:09.034 INFO:tasks.rados.rados.0.smithi104.stdout:193: finishing rollback tid 2 to smithi10428818-17 2020-03-09T00:25:09.035 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 17 v 33 (ObjNum 77 snap 12 seq_num 77) dirty exists 2020-03-09T00:25:09.035 INFO:tasks.rados.rados.0.smithi104.stdout:189: finishing copy_from to smithi10428818-7 2020-03-09T00:25:09.035 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 7 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-03-09T00:25:09.035 INFO:tasks.rados.rados.0.smithi104.stdout:189: finishing copy_from racing read to smithi10428818-7 2020-03-09T00:25:09.035 INFO:tasks.rados.rados.0.smithi104.stdout:188: finishing rollback tid 2 to smithi10428818-31 2020-03-09T00:25:09.035 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 31 v 28 (ObjNum 64 snap 8 seq_num 64) dirty exists 2020-03-09T00:25:09.035 INFO:tasks.rados.rados.0.smithi104.stdout:192: finishing write tid 1 to smithi10428818-19 2020-03-09T00:25:09.036 INFO:tasks.rados.rados.0.smithi104.stdout:192: finishing write tid 2 to smithi10428818-19 2020-03-09T00:25:09.036 INFO:tasks.rados.rados.0.smithi104.stdout:192: finishing write tid 3 to smithi10428818-19 2020-03-09T00:25:09.036 INFO:tasks.rados.rados.0.smithi104.stdout:188: done (6 left) 2020-03-09T00:25:09.036 INFO:tasks.rados.rados.0.smithi104.stdout:189: done (5 left) 2020-03-09T00:25:09.036 INFO:tasks.rados.rados.0.smithi104.stdout:191: done (4 left) 2020-03-09T00:25:09.036 INFO:tasks.rados.rados.0.smithi104.stdout:193: done (3 left) 2020-03-09T00:25:09.036 INFO:tasks.rados.rados.0.smithi104.stdout:194: done (2 left) 2020-03-09T00:25:09.037 INFO:tasks.rados.rados.0.smithi104.stdout:195: copy_from oid 28 from oid 39 current snap is 14 2020-03-09T00:25:09.037 INFO:tasks.rados.rados.0.smithi104.stdout:192: finishing write tid 5 to smithi10428818-19 2020-03-09T00:25:09.037 INFO:tasks.rados.rados.0.smithi104.stdout:190: finishing write tid 1 to smithi10428818-3 2020-03-09T00:25:09.037 INFO:tasks.rados.rados.0.smithi104.stdout:192: finishing write tid 6 to smithi10428818-19 2020-03-09T00:25:09.037 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 19 v 58 (ObjNum 93 snap 14 seq_num 93) dirty exists 2020-03-09T00:25:09.037 INFO:tasks.rados.rados.0.smithi104.stdout:192: left oid 19 (ObjNum 93 snap 14 seq_num 93) 2020-03-09T00:25:09.037 INFO:tasks.rados.rados.0.smithi104.stdout:190: finishing write tid 2 to smithi10428818-3 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:190: finishing write tid 3 to smithi10428818-3 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:190: finishing write tid 5 to smithi10428818-3 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:190: finishing write tid 6 to smithi10428818-3 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 3 v 48 (ObjNum 92 snap 14 seq_num 92) dirty exists 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:190: left oid 3 (ObjNum 92 snap 14 seq_num 92) 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:190: done (2 left) 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:192: done (1 left) 2020-03-09T00:25:09.038 INFO:tasks.rados.rados.0.smithi104.stdout:196: snap_remove snap 12 2020-03-09T00:25:09.039 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:09.030+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:09.133 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:09.132+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:09.155 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:09.155 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:09.155 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:09.155 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:09.163 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:25:09.234 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:09.233+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:09.278 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:09.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:09.278 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:09.278 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:09.279 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:09.286 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:25:09.335 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:09.334+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:09.403 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:09.404 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:09.404 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:09.404 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:09.404 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.404 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:185 3.es0 3:77c8bc20:::smithi10428818-47:4 [read 0~786432,checksum] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:25:09.404 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.811782+0000", 2020-03-09T00:25:09.404 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.589348365999999, 2020-03-09T00:25:09.405 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015338732000000001, 2020-03-09T00:25:09.405 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.405 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:09.405 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.405 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.405 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.406 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 185 2020-03-09T00:25:09.406 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.406 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.406 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.406 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811782+0000", 2020-03-09T00:25:09.406 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811782+0000", 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7703999999999997e-05 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.407 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.408 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811820+0000", 2020-03-09T00:25:09.408 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5599999999999998e-06 2020-03-09T00:25:09.408 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.408 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.408 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.408 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811823+0000", 2020-03-09T00:25:09.408 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3010000000000003e-06 2020-03-09T00:25:09.409 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.409 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.409 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.409 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811830+0000", 2020-03-09T00:25:09.409 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9929999999999997e-06 2020-03-09T00:25:09.409 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.409 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.410 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.410 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811834+0000", 2020-03-09T00:25:09.410 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0899000000000003e-05 2020-03-09T00:25:09.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.410 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.410 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.411 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811874+0000", 2020-03-09T00:25:09.411 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9910000000000008e-06 2020-03-09T00:25:09.411 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.411 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.411 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:09.411 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811883+0000", 2020-03-09T00:25:09.411 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028042219999999999 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814688+0000", 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3833999999999998e-05 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.412 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.413 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.814762+0000", 2020-03-09T00:25:09.413 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012359228 2020-03-09T00:25:09.413 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.413 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.413 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.413 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.827121+0000", 2020-03-09T00:25:09.413 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015338732000000001 2020-03-09T00:25:09.414 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.414 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.414 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.414 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.414 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.414 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:186 3.es0 3:77c8bc20:::smithi10428818-47:4 [read 0~786432,checksum] snapc 0=[] ondisk+read+known_if_redirected e38)", 2020-03-09T00:25:09.414 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:47.811886+0000", 2020-03-09T00:25:09.415 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.589244695000001, 2020-03-09T00:25:09.415 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015779517999999999, 2020-03-09T00:25:09.415 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.415 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:09.415 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.415 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.415 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.416 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 186 2020-03-09T00:25:09.416 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.416 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.416 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.416 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.416 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811886+0000", 2020-03-09T00:25:09.416 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.417 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.417 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.417 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.417 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811886+0000", 2020-03-09T00:25:09.417 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4892000000000001e-05 2020-03-09T00:25:09.417 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.417 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.418 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.418 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811951+0000", 2020-03-09T00:25:09.418 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5570000000000001e-06 2020-03-09T00:25:09.418 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.418 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.418 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.419 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811957+0000", 2020-03-09T00:25:09.419 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2e-05 2020-03-09T00:25:09.419 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.419 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.419 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.419 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811969+0000", 2020-03-09T00:25:09.419 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7720000000000002e-06 2020-03-09T00:25:09.420 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.420 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.420 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.811974+0000", 2020-03-09T00:25:09.420 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3767999999999998e-05 2020-03-09T00:25:09.420 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812018+0000", 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5399999999999996e-06 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.812026+0000", 2020-03-09T00:25:09.421 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003166079 2020-03-09T00:25:09.422 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.422 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.422 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.422 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815192+0000", 2020-03-09T00:25:09.422 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000126334 2020-03-09T00:25:09.422 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.422 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.423 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.423 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.815319+0000", 2020-03-09T00:25:09.423 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012346576 2020-03-09T00:25:09.423 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.423 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.423 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.424 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:47.827665+0000", 2020-03-09T00:25:09.424 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015779517999999999 2020-03-09T00:25:09.424 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.424 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.424 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.424 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.425 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:242 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~638976] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:09.425 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:09.425 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.554722081, 2020-03-09T00:25:09.425 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001, 2020-03-09T00:25:09.425 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.425 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.425 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.426 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.426 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.426 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 242 2020-03-09T00:25:09.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.426 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.426 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012420980000000001 2020-03-09T00:25:09.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.428 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.428 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847650+0000", 2020-03-09T00:25:09.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2689999999999996e-06 2020-03-09T00:25:09.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.428 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.428 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.429 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847657+0000", 2020-03-09T00:25:09.429 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2094e-05 2020-03-09T00:25:09.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.429 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.429 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847669+0000", 2020-03-09T00:25:09.430 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2690000000000002e-06 2020-03-09T00:25:09.430 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.430 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.430 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.430 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847676+0000", 2020-03-09T00:25:09.430 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3765999999999999e-05 2020-03-09T00:25:09.431 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.431 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.431 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.431 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847740+0000", 2020-03-09T00:25:09.431 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000155716 2020-03-09T00:25:09.431 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.431 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847896+0000", 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020523669999999998 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849948+0000", 2020-03-09T00:25:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010549378999999999 2020-03-09T00:25:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860497+0000", 2020-03-09T00:25:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046388970000000003 2020-03-09T00:25:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.434 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.434 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865136+0000", 2020-03-09T00:25:09.434 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2514000000000005e-05 2020-03-09T00:25:09.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.434 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.434 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.434 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865199+0000", 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:243 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~376832] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:09.435 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:09.436 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.552216278, 2020-03-09T00:25:09.436 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999, 2020-03-09T00:25:09.436 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.436 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.436 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.436 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.436 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.437 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 243 2020-03-09T00:25:09.437 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.437 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.437 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.437 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.437 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:09.437 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.438 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.438 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:09.438 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012192539999999999 2020-03-09T00:25:09.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850133+0000", 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1730000000000001e-06 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850143+0000", 2020-03-09T00:25:09.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1009000000000001e-05 2020-03-09T00:25:09.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.440 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.440 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850174+0000", 2020-03-09T00:25:09.440 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0500000000000003e-06 2020-03-09T00:25:09.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850181+0000", 2020-03-09T00:25:09.441 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047873150000000003 2020-03-09T00:25:09.441 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.441 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.854968+0000", 2020-03-09T00:25:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013640000000000001 2020-03-09T00:25:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855104+0000", 2020-03-09T00:25:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001096669 2020-03-09T00:25:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.443 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.443 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.443 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856201+0000", 2020-03-09T00:25:09.443 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042977170000000004 2020-03-09T00:25:09.443 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.443 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.444 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.444 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860499+0000", 2020-03-09T00:25:09.444 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.008534827 2020-03-09T00:25:09.444 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.444 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.444 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.444 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869034+0000", 2020-03-09T00:25:09.445 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4033e-05 2020-03-09T00:25:09.445 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.445 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.445 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.445 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869118+0000", 2020-03-09T00:25:09.445 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999 2020-03-09T00:25:09.445 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.446 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.446 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.446 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.446 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.446 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:244 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (56) in=7b] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:09.446 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:09.447 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.550871059, 2020-03-09T00:25:09.447 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999, 2020-03-09T00:25:09.447 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.447 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.447 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.447 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 244 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:09.448 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.449 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.449 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.449 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:09.449 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000146107 2020-03-09T00:25:09.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.449 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.450 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.450 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850405+0000", 2020-03-09T00:25:09.450 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2099999999999999e-05 2020-03-09T00:25:09.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.450 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.450 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.450 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850418+0000", 2020-03-09T00:25:09.451 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9311000000000001e-05 2020-03-09T00:25:09.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.451 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.451 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.451 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850447+0000", 2020-03-09T00:25:09.451 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6460000000000004e-06 2020-03-09T00:25:09.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.452 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850453+0000", 2020-03-09T00:25:09.452 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011322660999999999 2020-03-09T00:25:09.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861776+0000", 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015394900000000001 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861930+0000", 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076229599999999998 2020-03-09T00:25:09.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.454 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.454 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862692+0000", 2020-03-09T00:25:09.454 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002593092 2020-03-09T00:25:09.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.454 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865285+0000", 2020-03-09T00:25:09.455 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039673670000000003 2020-03-09T00:25:09.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869253+0000", 2020-03-09T00:25:09.455 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010661 2020-03-09T00:25:09.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.456 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.456 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.456 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869359+0000", 2020-03-09T00:25:09.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999 2020-03-09T00:25:09.456 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.457 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.457 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.457 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.457 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:245 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:09.457 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:09.458 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.550624824, 2020-03-09T00:25:09.458 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998, 2020-03-09T00:25:09.458 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.458 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:09.458 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.458 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.459 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.459 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 245 2020-03-09T00:25:09.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.459 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.459 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.459 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2926999999999997e-05 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850548+0000", 2020-03-09T00:25:09.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7159999999999997e-06 2020-03-09T00:25:09.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850555+0000", 2020-03-09T00:25:09.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2537999999999999e-05 2020-03-09T00:25:09.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.462 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.462 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850568+0000", 2020-03-09T00:25:09.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6200000000000004e-06 2020-03-09T00:25:09.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850573+0000", 2020-03-09T00:25:09.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013277631 2020-03-09T00:25:09.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863851+0000", 2020-03-09T00:25:09.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016975200000000001 2020-03-09T00:25:09.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:09.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864021+0000", 2020-03-09T00:25:09.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058362869999999999 2020-03-09T00:25:09.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869857+0000", 2020-03-09T00:25:09.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011197 2020-03-09T00:25:09.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869969+0000", 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0016289570000000001 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.871598+0000", 2020-03-09T00:25:09.466 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998 2020-03-09T00:25:09.467 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.467 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.467 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.467 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:289 3.es0 3:73be088e:::smithi10428818-42:head [append 0~737280] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:09.467 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:09.468 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.586652588, 2020-03-09T00:25:09.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001, 2020-03-09T00:25:09.468 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.468 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.468 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.468 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.468 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.469 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289 2020-03-09T00:25:09.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.469 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.469 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:09.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.470 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.470 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:09.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001180787 2020-03-09T00:25:09.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815659+0000", 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5960000000000004e-06 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815667+0000", 2020-03-09T00:25:09.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7425e-05 2020-03-09T00:25:09.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.472 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.472 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815685+0000", 2020-03-09T00:25:09.472 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0162e-05 2020-03-09T00:25:09.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.473 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.473 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815695+0000", 2020-03-09T00:25:09.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2490000000000004e-05 2020-03-09T00:25:09.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.473 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.473 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815787+0000", 2020-03-09T00:25:09.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7851999999999998e-05 2020-03-09T00:25:09.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815885+0000", 2020-03-09T00:25:09.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001491497 2020-03-09T00:25:09.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817377+0000", 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010162513999999999 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827539+0000", 2020-03-09T00:25:09.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027651400000000001 2020-03-09T00:25:09.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830304+0000", 2020-03-09T00:25:09.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8808999999999995e-05 2020-03-09T00:25:09.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830403+0000", 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.477 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.478 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.478 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:291 3.es0 3:73be088e:::smithi10428818-42:head [setxattr _header (56) in=7b] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:09.478 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.817611+0000", 2020-03-09T00:25:09.478 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.58351944, 2020-03-09T00:25:09.478 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015387929, 2020-03-09T00:25:09.478 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.478 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 291 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.479 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.480 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817611+0000", 2020-03-09T00:25:09.480 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.480 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.480 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.480 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.480 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817611+0000", 2020-03-09T00:25:09.480 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0197000000000004e-05 2020-03-09T00:25:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817701+0000", 2020-03-09T00:25:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5245000000000001e-05 2020-03-09T00:25:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817716+0000", 2020-03-09T00:25:09.482 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3361e-05 2020-03-09T00:25:09.482 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.482 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817730+0000", 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5099999999999998e-06 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817735+0000", 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010148786999999999 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827884+0000", 2020-03-09T00:25:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00026579300000000001 2020-03-09T00:25:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.828150+0000", 2020-03-09T00:25:09.485 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00088932599999999996 2020-03-09T00:25:09.485 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.485 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.485 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.485 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.829039+0000", 2020-03-09T00:25:09.485 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018813409999999999 2020-03-09T00:25:09.485 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830920+0000", 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001950266 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.486 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.832871+0000", 2020-03-09T00:25:09.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012810299999999999 2020-03-09T00:25:09.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.487 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.487 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.487 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.832999+0000", 2020-03-09T00:25:09.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015387929 2020-03-09T00:25:09.487 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.488 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.488 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.488 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.488 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:292 3.es0 3:73be088e:::smithi10428818-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:09.488 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:09.488 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.583349565000001, 2020-03-09T00:25:09.489 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999, 2020-03-09T00:25:09.489 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.489 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:09.489 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.489 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.489 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.489 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 292 2020-03-09T00:25:09.490 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.490 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.490 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.490 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.490 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:09.490 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7518000000000001e-05 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.491 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.492 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817828+0000", 2020-03-09T00:25:09.492 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:09.492 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.492 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.492 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.492 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817834+0000", 2020-03-09T00:25:09.492 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2721000000000001e-05 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817847+0000", 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2339999999999996e-06 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.493 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.494 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817851+0000", 2020-03-09T00:25:09.494 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012178102999999999 2020-03-09T00:25:09.494 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.494 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.494 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.494 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830029+0000", 2020-03-09T00:25:09.494 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012795100000000001 2020-03-09T00:25:09.495 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.495 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.495 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:09.495 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830157+0000", 2020-03-09T00:25:09.495 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003501414 2020-03-09T00:25:09.495 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.495 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.496 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.496 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833659+0000", 2020-03-09T00:25:09.496 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5081999999999996e-05 2020-03-09T00:25:09.496 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.496 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.496 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.496 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833744+0000", 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002393077 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.836137+0000", 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.497 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.498 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.498 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:339 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:09.498 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:09.498 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 6.593441994, 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998, 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 339 2020-03-09T00:25:09.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.500 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:09.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00074630200000000001 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808435+0000", 2020-03-09T00:25:09.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6087e-05 2020-03-09T00:25:09.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.502 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.502 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808451+0000", 2020-03-09T00:25:09.502 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.413e-05 2020-03-09T00:25:09.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808465+0000", 2020-03-09T00:25:09.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4320000000000004e-06 2020-03-09T00:25:09.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808471+0000", 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046408889999999996 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813112+0000", 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013929799999999999 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813252+0000", 2020-03-09T00:25:09.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017257030000000001 2020-03-09T00:25:09.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.814977+0000", 2020-03-09T00:25:09.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099845400000000001 2020-03-09T00:25:09.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.824962+0000", 2020-03-09T00:25:09.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049262399999999999 2020-03-09T00:25:09.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825454+0000", 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.462e-05 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.507 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825539+0000", 2020-03-09T00:25:09.508 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998 2020-03-09T00:25:09.508 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.508 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.508 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.508 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.508 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:340 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~294912] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:09.509 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:09.509 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 6.5908375660000003, 2020-03-09T00:25:09.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244, 2020-03-09T00:25:09.509 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.509 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.509 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.509 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 340 2020-03-09T00:25:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:09.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:09.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000216388 2020-03-09T00:25:09.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810509+0000", 2020-03-09T00:25:09.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6459999999999998e-06 2020-03-09T00:25:09.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.513 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810515+0000", 2020-03-09T00:25:09.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.799e-06 2020-03-09T00:25:09.513 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.513 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.513 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.513 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810524+0000", 2020-03-09T00:25:09.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:09.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.514 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810528+0000", 2020-03-09T00:25:09.514 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007805228 2020-03-09T00:25:09.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818333+0000", 2020-03-09T00:25:09.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013953500000000001 2020-03-09T00:25:09.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818473+0000", 2020-03-09T00:25:09.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00094954999999999996 2020-03-09T00:25:09.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.819423+0000", 2020-03-09T00:25:09.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056089679999999998 2020-03-09T00:25:09.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825032+0000", 2020-03-09T00:25:09.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015242280000000001 2020-03-09T00:25:09.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.518 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826556+0000", 2020-03-09T00:25:09.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4308000000000007e-05 2020-03-09T00:25:09.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.518 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.518 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826640+0000", 2020-03-09T00:25:09.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244 2020-03-09T00:25:09.519 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.519 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.519 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.519 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:341 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 6.5905724829999999, 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001, 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.520 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 341 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:09.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6538999999999997e-05 2020-03-09T00:25:09.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810594+0000", 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3629999999999998e-06 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810598+0000", 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1715e-05 2020-03-09T00:25:09.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.524 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810610+0000", 2020-03-09T00:25:09.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7019999999999999e-06 2020-03-09T00:25:09.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810613+0000", 2020-03-09T00:25:09.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016583064000000002 2020-03-09T00:25:09.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827196+0000", 2020-03-09T00:25:09.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000134802 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827331+0000", 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079633999999999998 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828127+0000", 2020-03-09T00:25:09.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000914331 2020-03-09T00:25:09.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.527 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.829042+0000", 2020-03-09T00:25:09.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045226199999999998 2020-03-09T00:25:09.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:09.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000101851 2020-03-09T00:25:09.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833666+0000", 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.530 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:342 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:09.530 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:09.530 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 6.5904943820000002, 2020-03-09T00:25:09.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001, 2020-03-09T00:25:09.530 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.531 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:09.531 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.531 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.531 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.531 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 342 2020-03-09T00:25:09.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.531 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:09.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1574e-05 2020-03-09T00:25:09.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.533 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810658+0000", 2020-03-09T00:25:09.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.196e-06 2020-03-09T00:25:09.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810661+0000", 2020-03-09T00:25:09.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6000000000000003e-06 2020-03-09T00:25:09.534 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.535 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810667+0000", 2020-03-09T00:25:09.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3269999999999999e-06 2020-03-09T00:25:09.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.535 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810670+0000", 2020-03-09T00:25:09.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018222015000000001 2020-03-09T00:25:09.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.536 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828892+0000", 2020-03-09T00:25:09.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107623 2020-03-09T00:25:09.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828999+0000", 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005589938 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834589+0000", 2020-03-09T00:25:09.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012716299999999999 2020-03-09T00:25:09.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834716+0000", 2020-03-09T00:25:09.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048828580000000003 2020-03-09T00:25:09.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.839599+0000", 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.540 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:374 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:09.540 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:09.540 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 5.6069875590000002, 2020-03-09T00:25:09.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001, 2020-03-09T00:25:09.540 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.540 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:09.540 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 374 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:09.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:09.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3215999999999998e-05 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794216+0000", 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7970000000000003e-06 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.543 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794223+0000", 2020-03-09T00:25:09.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2177e-05 2020-03-09T00:25:09.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794235+0000", 2020-03-09T00:25:09.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8439999999999997e-06 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794240+0000", 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7449999999999997e-05 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794337+0000", 2020-03-09T00:25:09.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0590000000000003e-06 2020-03-09T00:25:09.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:09.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794345+0000", 2020-03-09T00:25:09.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032359939999999998 2020-03-09T00:25:09.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797581+0000", 2020-03-09T00:25:09.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114491 2020-03-09T00:25:09.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797696+0000", 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016180261000000001 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.813876+0000", 2020-03-09T00:25:09.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001 2020-03-09T00:25:09.549 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.549 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.549 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.549 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:433 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~671744] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:09.549 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5999536330000002, 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002, 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.550 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 433 2020-03-09T00:25:09.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.551 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.551 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:09.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030767469999999999 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804253+0000", 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1990000000000006e-06 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804262+0000", 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9800999999999999e-05 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804281+0000", 2020-03-09T00:25:09.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0343000000000001e-05 2020-03-09T00:25:09.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804292+0000", 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000220564 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804512+0000", 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012449100000000001 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804637+0000", 2020-03-09T00:25:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00145191 2020-03-09T00:25:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806089+0000", 2020-03-09T00:25:09.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020643070999999999 2020-03-09T00:25:09.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826732+0000", 2020-03-09T00:25:09.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081151190000000005 2020-03-09T00:25:09.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834847+0000", 2020-03-09T00:25:09.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1645999999999996e-05 2020-03-09T00:25:09.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834939+0000", 2020-03-09T00:25:09.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002 2020-03-09T00:25:09.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.559 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=66, reqid=client.4751.0:397, at_version=55'17, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5967661279999996, 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001, 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.560 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:09.561 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.561 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:09.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104311 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804469+0000", 2020-03-09T00:25:09.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5208999999999999e-05 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804494+0000", 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5976000000000001e-05 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804510+0000", 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0229999999999998e-06 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804516+0000", 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057666139999999998 2020-03-09T00:25:09.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810282+0000", 2020-03-09T00:25:09.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3182e-05 2020-03-09T00:25:09.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810296+0000", 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016691906999999999 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826987+0000", 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8390000000000003e-05 2020-03-09T00:25:09.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827036+0000", 2020-03-09T00:25:09.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001 2020-03-09T00:25:09.567 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.567 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.567 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.568 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:434 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~770048] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:09.568 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:09.568 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5960243590000003, 2020-03-09T00:25:09.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003, 2020-03-09T00:25:09.568 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.569 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.569 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.569 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.569 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.569 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 434 2020-03-09T00:25:09.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.569 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:09.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:09.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011664830000000001 2020-03-09T00:25:09.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806272+0000", 2020-03-09T00:25:09.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9829999999999999e-06 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806279+0000", 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1960000000000001e-05 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806291+0000", 2020-03-09T00:25:09.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4999999999999999e-06 2020-03-09T00:25:09.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806297+0000", 2020-03-09T00:25:09.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075845660000000001 2020-03-09T00:25:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.813882+0000", 2020-03-09T00:25:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013610299999999999 2020-03-09T00:25:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.814018+0000", 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047231749999999996 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.818741+0000", 2020-03-09T00:25:09.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01050245 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829243+0000", 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0085708439999999993 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837814+0000", 2020-03-09T00:25:09.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4356000000000003e-05 2020-03-09T00:25:09.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837898+0000", 2020-03-09T00:25:09.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003 2020-03-09T00:25:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:435 3.es0 3:73e16f4f:::smithi10428818-13:head [setxattr _header (57) in=7b] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5947785850000002, 2020-03-09T00:25:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003, 2020-03-09T00:25:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.579 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.580 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 435 2020-03-09T00:25:09.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.580 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:09.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4781000000000001e-05 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806427+0000", 2020-03-09T00:25:09.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9479999999999996e-06 2020-03-09T00:25:09.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806433+0000", 2020-03-09T00:25:09.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1838000000000001e-05 2020-03-09T00:25:09.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806444+0000", 2020-03-09T00:25:09.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7199999999999997e-06 2020-03-09T00:25:09.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806449+0000", 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020656463999999999 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827106+0000", 2020-03-09T00:25:09.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013190700000000001 2020-03-09T00:25:09.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.585 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.585 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827237+0000", 2020-03-09T00:25:09.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00078356200000000002 2020-03-09T00:25:09.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828021+0000", 2020-03-09T00:25:09.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001223678 2020-03-09T00:25:09.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829245+0000", 2020-03-09T00:25:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087850050000000002 2020-03-09T00:25:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838030+0000", 2020-03-09T00:25:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7771999999999998e-05 2020-03-09T00:25:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838127+0000", 2020-03-09T00:25:09.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003 2020-03-09T00:25:09.588 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.588 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.588 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.589 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:436 3.es0 3:73e16f4f:::smithi10428818-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-03-09T00:25:09.589 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:09.589 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5946420909999999, 2020-03-09T00:25:09.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998, 2020-03-09T00:25:09.589 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.590 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:09.590 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:09.590 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:09.590 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.590 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 436 2020-03-09T00:25:09.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:09.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3732000000000001e-05 2020-03-09T00:25:09.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806532+0000", 2020-03-09T00:25:09.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5879999999999998e-06 2020-03-09T00:25:09.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806541+0000", 2020-03-09T00:25:09.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2738000000000001e-05 2020-03-09T00:25:09.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806553+0000", 2020-03-09T00:25:09.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.605e-06 2020-03-09T00:25:09.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806558+0000", 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021728470999999999 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828286+0000", 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113791 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:09.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828400+0000", 2020-03-09T00:25:09.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010563283 2020-03-09T00:25:09.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838963+0000", 2020-03-09T00:25:09.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121599 2020-03-09T00:25:09.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.597 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:09.597 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.839085+0000", 2020-03-09T00:25:09.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028128480000000002 2020-03-09T00:25:09.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.598 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.598 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.841898+0000", 2020-03-09T00:25:09.598 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998 2020-03-09T00:25:09.598 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.598 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.598 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.599 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=67, reqid=client.4751.0:398, at_version=55'18, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:09.599 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:09.599 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5907513169999996, 2020-03-09T00:25:09.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999, 2020-03-09T00:25:09.599 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:09.600 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:09.600 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:09.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:09.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:09.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:09.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:09.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:09.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000127076 2020-03-09T00:25:09.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:09.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810506+0000", 2020-03-09T00:25:09.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3247999999999998e-05 2020-03-09T00:25:09.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.602 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:09.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810529+0000", 2020-03-09T00:25:09.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4992e-05 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810544+0000", 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9669999999999998e-06 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:09.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810550+0000", 2020-03-09T00:25:09.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098081420000000006 2020-03-09T00:25:09.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:09.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820358+0000", 2020-03-09T00:25:09.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5410999999999999e-05 2020-03-09T00:25:09.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:09.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820374+0000", 2020-03-09T00:25:09.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066733160000000003 2020-03-09T00:25:09.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827047+0000", 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4610999999999998e-05 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827082+0000", 2020-03-09T00:25:09.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999 2020-03-09T00:25:09.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.607 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:09.607 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:09.607 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:09.608 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:09.610 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:09.608+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:09.610 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:25:09.710 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:09.708+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:09.731 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:09.731 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:09.731 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:09.731 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:09.739 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:25:09.780 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:09.801 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:09.810 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:09.808+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:09.816 INFO:tasks.rados.rados.0.smithi104.stdout:196: done (1 left) 2020-03-09T00:25:09.816 INFO:tasks.rados.rados.0.smithi104.stdout:197: delete oid 3 current snap is 14 2020-03-09T00:25:09.816 INFO:tasks.rados.rados.0.smithi104.stdout:195: finishing copy_from to smithi10428818-28 2020-03-09T00:25:09.816 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 28 v 54 (ObjNum 89 snap 14 seq_num 89) dirty exists 2020-03-09T00:25:09.817 INFO:tasks.rados.rados.0.smithi104.stdout:195: finishing copy_from racing read to smithi10428818-28 2020-03-09T00:25:09.824 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is unset 2020-03-09T00:25:09.833 INFO:tasks.rados.rados.0.smithi104.stdout:195: done (1 left) 2020-03-09T00:25:09.833 INFO:tasks.rados.rados.0.smithi104.stdout:197: done (0 left) 2020-03-09T00:25:09.833 INFO:tasks.rados.rados.0.smithi104.stdout:198: setattr oid 46 current snap is 14 2020-03-09T00:25:09.833 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:09.835 INFO:tasks.rados.rados.0.smithi104.stdout:199: snap_create 2020-03-09T00:25:09.842 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 46 v 47 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-03-09T00:25:09.856 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:09.864 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:09.864 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:09.864 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:09.865 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:09.865 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:09.873 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:25:09.876 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:09.898 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:09.911 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:09.909+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:09.917 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:25:09.937 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:09.956 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:09.977 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:09.991 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:09.991 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:09.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:193 3.fs0 3:fa16d489:::smithi10428818-27:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811772+0000", 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.176637640999999, 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012182698, 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 193 2020-03-09T00:25:09.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:09.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811772+0000", 2020-03-09T00:25:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811772+0000", 2020-03-09T00:25:09.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8804000000000001e-05 2020-03-09T00:25:09.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:09.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:09.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:09.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811800+0000", 2020-03-09T00:25:09.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0749999999999997e-06 2020-03-09T00:25:09.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:09.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811805+0000", 2020-03-09T00:25:10.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1240000000000002e-06 2020-03-09T00:25:10.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811812+0000", 2020-03-09T00:25:10.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9950000000000001e-06 2020-03-09T00:25:10.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811815+0000", 2020-03-09T00:25:10.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059357740000000004 2020-03-09T00:25:10.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817750+0000", 2020-03-09T00:25:10.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000111483 2020-03-09T00:25:10.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.016 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.817862+0000", 2020-03-09T00:25:10.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00073629399999999997 2020-03-09T00:25:10.016 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.016 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.818598+0000", 2020-03-09T00:25:10.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001344425 2020-03-09T00:25:10.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819943+0000", 2020-03-09T00:25:10.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003966832 2020-03-09T00:25:10.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823909+0000", 2020-03-09T00:25:10.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4892000000000003e-05 2020-03-09T00:25:10.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.823954+0000", 2020-03-09T00:25:10.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012182698 2020-03-09T00:25:10.021 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.022 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.022 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.022 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:194 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:10.022 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.811838+0000", 2020-03-09T00:25:10.023 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.176570748, 2020-03-09T00:25:10.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012302923, 2020-03-09T00:25:10.023 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.023 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.023 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.023 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.023 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.024 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 194 2020-03-09T00:25:10.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.024 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811838+0000", 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811838+0000", 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3748999999999999e-05 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811862+0000", 2020-03-09T00:25:10.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.455e-06 2020-03-09T00:25:10.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811866+0000", 2020-03-09T00:25:10.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4569999999999999e-06 2020-03-09T00:25:10.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.029 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811873+0000", 2020-03-09T00:25:10.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7450000000000001e-06 2020-03-09T00:25:10.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.811877+0000", 2020-03-09T00:25:10.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007146693 2020-03-09T00:25:10.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819024+0000", 2020-03-09T00:25:10.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000104384 2020-03-09T00:25:10.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819128+0000", 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052495999999999999 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819653+0000", 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000337096 2020-03-09T00:25:10.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.819990+0000", 2020-03-09T00:25:10.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040765749999999998 2020-03-09T00:25:10.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.824067+0000", 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4808999999999995e-05 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.824141+0000", 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012302923 2020-03-09T00:25:10.036 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.037 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.037 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.037 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:299 3.fs0 3:f75fa7c5:::smithi10428818-15:head [copy-from ver 4] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:10.037 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:10.037 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.170294171, 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020952168, 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 299 2020-03-09T00:25:10.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.039 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:10.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012868800000000001 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818244+0000", 2020-03-09T00:25:10.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5429999999999994e-06 2020-03-09T00:25:10.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818252+0000", 2020-03-09T00:25:10.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1410999999999999e-05 2020-03-09T00:25:10.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818314+0000", 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9577000000000002e-05 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818343+0000", 2020-03-09T00:25:10.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8003e-05 2020-03-09T00:25:10.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818401+0000", 2020-03-09T00:25:10.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014568900000000001 2020-03-09T00:25:10.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818547+0000", 2020-03-09T00:25:10.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0078751659999999994 2020-03-09T00:25:10.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.826422+0000", 2020-03-09T00:25:10.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023378689999999998 2020-03-09T00:25:10.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828760+0000", 2020-03-09T00:25:10.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010241317999999999 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839001+0000", 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5903999999999998e-05 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839067+0000", 2020-03-09T00:25:10.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020952168 2020-03-09T00:25:10.047 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.047 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.047 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:300 3.fs0 3:f75fa7c5:::smithi10428818-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.169975452999999, 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021129146000000001, 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.048 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.049 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.049 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 300 2020-03-09T00:25:10.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.049 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2379000000000005e-05 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818516+0000", 2020-03-09T00:25:10.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8039999999999998e-06 2020-03-09T00:25:10.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818522+0000", 2020-03-09T00:25:10.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.208e-05 2020-03-09T00:25:10.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818534+0000", 2020-03-09T00:25:10.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5580000000000002e-06 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818539+0000", 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060248300000000001 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.819141+0000", 2020-03-09T00:25:10.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1888e-05 2020-03-09T00:25:10.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.054 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:10.054 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.819213+0000", 2020-03-09T00:25:10.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091353879999999995 2020-03-09T00:25:10.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828348+0000", 2020-03-09T00:25:10.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012281999999999999 2020-03-09T00:25:10.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828471+0000", 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010931410000000001 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839403+0000", 2020-03-09T00:25:10.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3260000000000001e-05 2020-03-09T00:25:10.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839466+0000", 2020-03-09T00:25:10.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7076000000000004e-05 2020-03-09T00:25:10.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839563+0000", 2020-03-09T00:25:10.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021129146000000001 2020-03-09T00:25:10.058 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.058 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.058 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.059 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:301 3.cs0 3:31ded997:::smithi10428818-32:head [append 0~753664] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:10.059 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.828231+0000", 2020-03-09T00:25:10.059 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.160177984000001, 2020-03-09T00:25:10.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017638918999999999, 2020-03-09T00:25:10.059 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 301 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.060 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828231+0000", 2020-03-09T00:25:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828231+0000", 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00059794500000000001 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828829+0000", 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4290000000000007e-06 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828839+0000", 2020-03-09T00:25:10.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4939e-05 2020-03-09T00:25:10.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828864+0000", 2020-03-09T00:25:10.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3840000000000002e-06 2020-03-09T00:25:10.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828870+0000", 2020-03-09T00:25:10.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9761e-05 2020-03-09T00:25:10.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828920+0000", 2020-03-09T00:25:10.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012904799999999999 2020-03-09T00:25:10.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829049+0000", 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015728630000000001 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830622+0000", 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0071465979999999997 2020-03-09T00:25:10.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837768+0000", 2020-03-09T00:25:10.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080003439999999995 2020-03-09T00:25:10.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.845769+0000", 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010160799999999999 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.845870+0000", 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017638918999999999 2020-03-09T00:25:10.068 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.069 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.069 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.069 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:302 3.cs0 3:31ded997:::smithi10428818-32:head [append 0~376832] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:10.069 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.829345+0000", 2020-03-09T00:25:10.069 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.159064161, 2020-03-09T00:25:10.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016771899, 2020-03-09T00:25:10.070 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.070 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.070 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.070 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 302 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829345+0000", 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829345+0000", 2020-03-09T00:25:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035359600000000002 2020-03-09T00:25:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829699+0000", 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2979999999999992e-06 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829707+0000", 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5423000000000001e-05 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829722+0000", 2020-03-09T00:25:10.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5549999999999999e-06 2020-03-09T00:25:10.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829729+0000", 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004559034 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.834288+0000", 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000136699 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.834425+0000", 2020-03-09T00:25:10.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00070708999999999995 2020-03-09T00:25:10.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.835132+0000", 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002638488 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837770+0000", 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082902300000000009 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846060+0000", 2020-03-09T00:25:10.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6486000000000001e-05 2020-03-09T00:25:10.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846117+0000", 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016771899 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:303 3.cs0 3:31ded997:::smithi10428818-32:head [setxattr _header (56) in=7b] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:10.079 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.829801+0000", 2020-03-09T00:25:10.080 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.158608371, 2020-03-09T00:25:10.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016503021, 2020-03-09T00:25:10.080 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.080 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.080 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 303 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829801+0000", 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829801+0000", 2020-03-09T00:25:10.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4760999999999996e-05 2020-03-09T00:25:10.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829896+0000", 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0380000000000002e-06 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829903+0000", 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2405000000000001e-05 2020-03-09T00:25:10.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829915+0000", 2020-03-09T00:25:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2349999999999999e-06 2020-03-09T00:25:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829920+0000", 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069454039999999996 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.836866+0000", 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0329999999999998e-05 2020-03-09T00:25:10.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.836946+0000", 2020-03-09T00:25:10.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049675499999999998 2020-03-09T00:25:10.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837443+0000", 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00032799000000000001 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837771+0000", 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0084763660000000008 2020-03-09T00:25:10.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846247+0000", 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6737000000000002e-05 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846304+0000", 2020-03-09T00:25:10.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016503021 2020-03-09T00:25:10.089 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.089 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.089 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.089 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:304 3.cs0 3:31ded997:::smithi10428818-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.158443971000001, 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018218758000000002, 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.090 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.091 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 304 2020-03-09T00:25:10.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.091 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:10.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7318999999999999e-05 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830013+0000", 2020-03-09T00:25:10.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1289999999999998e-06 2020-03-09T00:25:10.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830020+0000", 2020-03-09T00:25:10.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2313000000000001e-05 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830032+0000", 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7829999999999998e-06 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830037+0000", 2020-03-09T00:25:10.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007629449 2020-03-09T00:25:10.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837666+0000", 2020-03-09T00:25:10.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1544000000000003e-05 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837758+0000", 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0089964079999999991 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846754+0000", 2020-03-09T00:25:10.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113248 2020-03-09T00:25:10.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846867+0000", 2020-03-09T00:25:10.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001316565 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.848184+0000", 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018218758000000002 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.098 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.099 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:344 3.fs0 3:f2571ca0:::smithi10428818-48:head [copy-from ver 35] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:10.099 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:10.099 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.176598061, 2020-03-09T00:25:10.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066389322000000001, 2020-03-09T00:25:10.099 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 344 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:10.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:10.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100208 2020-03-09T00:25:10.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811911+0000", 2020-03-09T00:25:10.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1372999999999999e-05 2020-03-09T00:25:10.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811923+0000", 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5517999999999999e-05 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811938+0000", 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7190000000000004e-06 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811945+0000", 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1340000000000006e-05 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812006+0000", 2020-03-09T00:25:10.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014919999999999999 2020-03-09T00:25:10.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812156+0000", 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.038728380999999999 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.850884+0000", 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013700324 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864584+0000", 2020-03-09T00:25:10.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013548961 2020-03-09T00:25:10.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878133+0000", 2020-03-09T00:25:10.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7298000000000005e-05 2020-03-09T00:25:10.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878200+0000", 2020-03-09T00:25:10.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066389322000000001 2020-03-09T00:25:10.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.109 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.109 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.109 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:345 3.fs0 3:f2571ca0:::smithi10428818-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:10.109 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.1764094790000001, 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066989273000000002, 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 345 2020-03-09T00:25:10.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.111 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:10.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:10.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5945000000000001e-05 2020-03-09T00:25:10.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812046+0000", 2020-03-09T00:25:10.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.545e-06 2020-03-09T00:25:10.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812052+0000", 2020-03-09T00:25:10.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3108999999999999e-05 2020-03-09T00:25:10.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812065+0000", 2020-03-09T00:25:10.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.566e-06 2020-03-09T00:25:10.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812070+0000", 2020-03-09T00:25:10.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00034552600000000003 2020-03-09T00:25:10.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812415+0000", 2020-03-09T00:25:10.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8084999999999997e-05 2020-03-09T00:25:10.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812474+0000", 2020-03-09T00:25:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.051585182 2020-03-09T00:25:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864059+0000", 2020-03-09T00:25:10.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8030999999999999e-05 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864137+0000", 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014720627 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878857+0000", 2020-03-09T00:25:10.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9045999999999997e-05 2020-03-09T00:25:10.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878916+0000", 2020-03-09T00:25:10.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2611e-05 2020-03-09T00:25:10.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878989+0000", 2020-03-09T00:25:10.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066989273000000002 2020-03-09T00:25:10.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.121 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.121 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.121 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:380 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~557056] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:10.122 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:10.122 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.176569507, 2020-03-09T00:25:10.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019518607, 2020-03-09T00:25:10.122 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.122 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.122 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.123 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.123 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.123 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 380 2020-03-09T00:25:10.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.123 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:10.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:10.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036718500000000002 2020-03-09T00:25:10.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812207+0000", 2020-03-09T00:25:10.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3760000000000008e-06 2020-03-09T00:25:10.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812215+0000", 2020-03-09T00:25:10.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5866999999999999e-05 2020-03-09T00:25:10.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812231+0000", 2020-03-09T00:25:10.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2020-03-09T00:25:10.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812238+0000", 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6171000000000001e-05 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812304+0000", 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8213000000000003e-05 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812402+0000", 2020-03-09T00:25:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012745790000000001 2020-03-09T00:25:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.813676+0000", 2020-03-09T00:25:10.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095836270000000008 2020-03-09T00:25:10.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.823260+0000", 2020-03-09T00:25:10.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080215159999999994 2020-03-09T00:25:10.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831282+0000", 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6711999999999993e-05 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831358+0000", 2020-03-09T00:25:10.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019518607 2020-03-09T00:25:10.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.132 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.132 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:381 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~671744] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:10.133 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.816921+0000", 2020-03-09T00:25:10.133 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1714881090000002, 2020-03-09T00:25:10.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014673837, 2020-03-09T00:25:10.133 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.133 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.133 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.133 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 381 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816921+0000", 2020-03-09T00:25:10.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816921+0000", 2020-03-09T00:25:10.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00043486499999999999 2020-03-09T00:25:10.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817356+0000", 2020-03-09T00:25:10.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9449999999999999e-06 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817365+0000", 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5274e-05 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817380+0000", 2020-03-09T00:25:10.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0170000000000002e-06 2020-03-09T00:25:10.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817386+0000", 2020-03-09T00:25:10.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00063899699999999998 2020-03-09T00:25:10.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818025+0000", 2020-03-09T00:25:10.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6817999999999994e-05 2020-03-09T00:25:10.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818112+0000", 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010584279999999999 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.819170+0000", 2020-03-09T00:25:10.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061751089999999998 2020-03-09T00:25:10.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.825346+0000", 2020-03-09T00:25:10.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0062014399999999999 2020-03-09T00:25:10.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831547+0000", 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7944e-05 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831595+0000", 2020-03-09T00:25:10.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014673837 2020-03-09T00:25:10.143 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.143 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.143 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.143 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:382 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~98304] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:10.143 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.817661+0000", 2020-03-09T00:25:10.144 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.170748262, 2020-03-09T00:25:10.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014104438, 2020-03-09T00:25:10.144 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.144 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.144 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.144 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.144 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.145 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 382 2020-03-09T00:25:10.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.145 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817661+0000", 2020-03-09T00:25:10.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817661+0000", 2020-03-09T00:25:10.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014479099999999999 2020-03-09T00:25:10.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817806+0000", 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9469999999999997e-06 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817813+0000", 2020-03-09T00:25:10.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2903e-05 2020-03-09T00:25:10.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817826+0000", 2020-03-09T00:25:10.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0700000000000003e-06 2020-03-09T00:25:10.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817832+0000", 2020-03-09T00:25:10.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056750000000000004 2020-03-09T00:25:10.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.823507+0000", 2020-03-09T00:25:10.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000145112 2020-03-09T00:25:10.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.823652+0000", 2020-03-09T00:25:10.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00093108100000000005 2020-03-09T00:25:10.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.824583+0000", 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033256219999999999 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.827908+0000", 2020-03-09T00:25:10.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037940669999999999 2020-03-09T00:25:10.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831703+0000", 2020-03-09T00:25:10.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2844999999999999e-05 2020-03-09T00:25:10.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831765+0000", 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014104438 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.155 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:383 3.fs0 3:fa16d489:::smithi10428818-27:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:10.155 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.817885+0000", 2020-03-09T00:25:10.155 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.170524264, 2020-03-09T00:25:10.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015305879, 2020-03-09T00:25:10.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 383 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817885+0000", 2020-03-09T00:25:10.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817885+0000", 2020-03-09T00:25:10.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012183 2020-03-09T00:25:10.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818007+0000", 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0110000000000003e-06 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818014+0000", 2020-03-09T00:25:10.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4558000000000001e-05 2020-03-09T00:25:10.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:10.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.809e-06 2020-03-09T00:25:10.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818043+0000", 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073639639999999998 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.825407+0000", 2020-03-09T00:25:10.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000265135 2020-03-09T00:25:10.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.825672+0000", 2020-03-09T00:25:10.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001730091 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.827402+0000", 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00055959 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.827962+0000", 2020-03-09T00:25:10.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051231289999999997 2020-03-09T00:25:10.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833085+0000", 2020-03-09T00:25:10.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010576199999999999 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833191+0000", 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015305879 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.165 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:384 3.fs0 3:fa16d489:::smithi10428818-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:10.165 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:10.165 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1703074649999996, 2020-03-09T00:25:10.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018476747000000002, 2020-03-09T00:25:10.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 384 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:10.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1008999999999999e-05 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818153+0000", 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2199999999999997e-06 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818159+0000", 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3512999999999999e-05 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818173+0000", 2020-03-09T00:25:10.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5650000000000003e-06 2020-03-09T00:25:10.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818177+0000", 2020-03-09T00:25:10.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01043207 2020-03-09T00:25:10.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.828609+0000", 2020-03-09T00:25:10.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216252 2020-03-09T00:25:10.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:10.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.828825+0000", 2020-03-09T00:25:10.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005000539 2020-03-09T00:25:10.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833826+0000", 2020-03-09T00:25:10.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9849000000000001e-05 2020-03-09T00:25:10.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833906+0000", 2020-03-09T00:25:10.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026727299999999999 2020-03-09T00:25:10.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.836579+0000", 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018476747000000002 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:391 3.fs0 3:f1702e73:::smithi10428818-39:10 [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818228+0000", 2020-03-09T00:25:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1701809340000002, 2020-03-09T00:25:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014033159, 2020-03-09T00:25:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 391 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818228+0000", 2020-03-09T00:25:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818228+0000", 2020-03-09T00:25:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2390000000000001e-05 2020-03-09T00:25:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818321+0000", 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4460000000000004e-06 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818328+0000", 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3721e-05 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818342+0000", 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.246e-06 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818348+0000", 2020-03-09T00:25:10.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010521838 2020-03-09T00:25:10.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.828870+0000", 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000137391 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.829007+0000", 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0032541269999999999 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.832261+0000", 2020-03-09T00:25:10.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014033159 2020-03-09T00:25:10.182 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.182 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.182 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.183 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:392 3.fs0 3:f1702e73:::smithi10428818-39:10 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:10.183 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818406+0000", 2020-03-09T00:25:10.183 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1700034199999996, 2020-03-09T00:25:10.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014529808999999999, 2020-03-09T00:25:10.183 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.184 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:10.184 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.184 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.184 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.184 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 392 2020-03-09T00:25:10.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.184 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818406+0000", 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818406+0000", 2020-03-09T00:25:10.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4600999999999999e-05 2020-03-09T00:25:10.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818480+0000", 2020-03-09T00:25:10.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6190000000000002e-06 2020-03-09T00:25:10.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818488+0000", 2020-03-09T00:25:10.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3458000000000001e-05 2020-03-09T00:25:10.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818501+0000", 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8960000000000002e-06 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818506+0000", 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010866918999999999 2020-03-09T00:25:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.829373+0000", 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019685899999999999 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.829570+0000", 2020-03-09T00:25:10.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033654570000000001 2020-03-09T00:25:10.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.832936+0000", 2020-03-09T00:25:10.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014529808999999999 2020-03-09T00:25:10.190 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:393 3.fs0 3:f1702e73:::smithi10428818-39:10 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 6.1698506420000001, 2020-03-09T00:25:10.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01890696, 2020-03-09T00:25:10.192 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.192 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:10.192 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.192 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.192 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.192 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 393 2020-03-09T00:25:10.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.193 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:10.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1644000000000003e-05 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818600+0000", 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8680000000000001e-06 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818604+0000", 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1230000000000002e-06 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818611+0000", 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2399999999999999e-06 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818614+0000", 2020-03-09T00:25:10.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015764628999999999 2020-03-09T00:25:10.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.834379+0000", 2020-03-09T00:25:10.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016986400000000001 2020-03-09T00:25:10.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.834549+0000", 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002916592 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.837466+0000", 2020-03-09T00:25:10.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01890696 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:427 3.cs0 3:3776f87d:::smithi10428818-35:head [append 0~655360] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:04.221021+0000", 2020-03-09T00:25:10.199 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.7673882020000002, 2020-03-09T00:25:10.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012973739999999999, 2020-03-09T00:25:10.200 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.200 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:10.200 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:10.200 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:10.200 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:10.200 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 427 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221021+0000", 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221021+0000", 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00028972699999999999 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221311+0000", 2020-03-09T00:25:10.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8129999999999996e-06 2020-03-09T00:25:10.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221319+0000", 2020-03-09T00:25:10.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6160000000000008e-06 2020-03-09T00:25:10.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221328+0000", 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0939999999999998e-06 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221333+0000", 2020-03-09T00:25:10.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9361999999999999e-05 2020-03-09T00:25:10.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221383+0000", 2020-03-09T00:25:10.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013403699999999999 2020-03-09T00:25:10.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:10.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.221517+0000", 2020-03-09T00:25:10.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001211595 2020-03-09T00:25:10.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.222728+0000", 2020-03-09T00:25:10.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0079744210000000006 2020-03-09T00:25:10.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.230703+0000", 2020-03-09T00:25:10.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003240803 2020-03-09T00:25:10.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:10.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.233944+0000", 2020-03-09T00:25:10.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1272e-05 2020-03-09T00:25:10.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.233995+0000", 2020-03-09T00:25:10.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012973739999999999 2020-03-09T00:25:10.212 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.212 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.212 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.213 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as6 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:10.213 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.914226+0000", 2020-03-09T00:25:10.213 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 1.0741835689999999, 2020-03-09T00:25:10.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016547493999999999, 2020-03-09T00:25:10.214 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:10.214 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:10.214 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:10.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:10.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914226+0000", 2020-03-09T00:25:10.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:10.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:10.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914226+0000", 2020-03-09T00:25:10.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000205804 2020-03-09T00:25:10.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:10.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914431+0000", 2020-03-09T00:25:10.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6405000000000001e-05 2020-03-09T00:25:10.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:10.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914468+0000", 2020-03-09T00:25:10.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3586000000000001e-05 2020-03-09T00:25:10.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:10.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914491+0000", 2020-03-09T00:25:10.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6900000000000004e-06 2020-03-09T00:25:10.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:10.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914501+0000", 2020-03-09T00:25:10.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00051807100000000005 2020-03-09T00:25:10.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:10.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.915019+0000", 2020-03-09T00:25:10.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5773999999999998e-05 2020-03-09T00:25:10.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:10.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.915035+0000", 2020-03-09T00:25:10.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015701802000000001 2020-03-09T00:25:10.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:10.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.930737+0000", 2020-03-09T00:25:10.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6362000000000002e-05 2020-03-09T00:25:10.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:10.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:10.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:10.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.930773+0000", 2020-03-09T00:25:10.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016547493999999999 2020-03-09T00:25:10.222 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.222 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.222 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.223 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:10.223 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:10.223 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:10.225 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:10.226 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:10.226 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:10.226 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:10.226 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:10.226 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:10.226 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:10.036+0000 7f5faaffd700 0 -- 172.21.15.148:0/202859895 >> [v2:172.21.15.50:6816/27631,v1:172.21.15.50:6817/27631] conn(0x7f5f9002c070 msgr2=0x7f5f9002e460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.227 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.228 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.229 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.229 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:10.229 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:25:10.230 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:10.223+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:10.325 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:10.324+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:10.356 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:10.356 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:10.365 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:25:10.427 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:10.425+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:10.480 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:10.480 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:10.489 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:25:10.527 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:10.526+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:10.604 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:10.604 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:10.612 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:25:10.628 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:10.627+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:10.729 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:10.728+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:10.749 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:10.749 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:10.759 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:25:10.817 INFO:tasks.rados.rados.0.smithi104.stdout:198: done (1 left) 2020-03-09T00:25:10.818 INFO:tasks.rados.rados.0.smithi104.stdout:199: done (0 left) 2020-03-09T00:25:10.818 INFO:tasks.rados.rados.0.smithi104.stdout:200: snap_create 2020-03-09T00:25:10.844 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:10.843+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:10.925 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:10.926 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:10.931 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:10.930+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:10.934 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:25:11.031 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:11.030+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:11.100 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:11.101 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:11.109 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:25:11.133 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:11.132+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:11.233 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:11.231+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:11.238 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:11.238 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:11.247 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:25:11.334 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:11.332+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:11.374 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 14, 15] out_osds: [13, 5] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:25:11.374 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:25:11.374 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2020-03-09T00:25:11.374 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2020-03-09T00:25:11.375 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2020-03-09T00:25:11.375 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2020-03-09T00:25:11.404 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:11.404 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:11.419 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:25:11.435 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:11.433+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2020-03-09T00:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:11.536 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:11.533+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:11.574 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:11.574 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:11.583 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:25:11.637 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:11.635+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:11.712 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:11.712 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:11.722 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:25:11.764 INFO:tasks.rados.rados.0.smithi104.stdout:200: done (0 left) 2020-03-09T00:25:11.765 INFO:tasks.rados.rados.0.smithi104.stdout:201: copy_from oid 23 from oid 20 current snap is 16 2020-03-09T00:25:11.765 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 37 current snap is 16 2020-03-09T00:25:11.765 INFO:tasks.rados.rados.0.smithi104.stdout:202: seq_num 96 ranges {606208=458752,1064960=688128,1753088=475136} 2020-03-09T00:25:11.766 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:11.763+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:11.770 INFO:tasks.rados.rados.0.smithi104.stdout:202: writing smithi10428818-37 from 606208 to 1064960 tid 1 2020-03-09T00:25:11.774 INFO:tasks.rados.rados.0.smithi104.stdout:202: writing smithi10428818-37 from 1064960 to 1753088 tid 2 2020-03-09T00:25:11.777 INFO:tasks.rados.rados.0.smithi104.stdout:202: writing smithi10428818-37 from 1753088 to 2228224 tid 3 2020-03-09T00:25:11.777 INFO:tasks.rados.rados.0.smithi104.stdout:203: snap_remove snap 2 2020-03-09T00:25:11.836 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-03-09T00:25:11.839 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:11.836+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:11.845 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:11.845 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:11.851 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:25:11.940 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:11.938+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:12.008 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:12.008 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:12.017 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:25:12.041 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:12.040+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:12.136 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:12.136 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:12.141 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:12.140+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:12.145 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:25:12.230 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:12.242 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:12.240+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:12.260 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:12.260 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:12.269 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:25:12.343 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:12.341+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:12.360 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:12.381 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:12.385 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:12.385 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:12.393 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:25:12.401 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:12.421 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:12.441 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:12.443 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:12.442+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:12.461 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:12.481 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:25:12.501 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:12.521 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:12.528 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:12.528 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:12.528 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:12.528 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:12.537 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:25:12.541 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:12.561 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:12.581 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:12.582 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:12.580+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:12.600 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:12.622 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:12.641 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:12.646 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:12.644+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:12.662 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:12.662 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:12.662 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:12.662 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:12.663 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:12.663 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:12.663 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.663 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.663 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.664 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.664 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.664 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.664 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.664 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.664 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.664 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.665 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.665 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.665 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.665 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.665 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.665 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:12.671 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:25:12.747 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:12.745+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:12.770 INFO:tasks.rados.rados.0.smithi104.stdout:203: done (2 left) 2020-03-09T00:25:12.770 INFO:tasks.rados.rados.0.smithi104.stdout:204: rollback oid 13 current snap is 16 2020-03-09T00:25:12.771 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 13 to 14 2020-03-09T00:25:12.771 INFO:tasks.rados.rados.0.smithi104.stdout:202: finishing write tid 1 to smithi10428818-37 2020-03-09T00:25:12.771 INFO:tasks.rados.rados.0.smithi104.stdout:202: finishing write tid 2 to smithi10428818-37 2020-03-09T00:25:12.771 INFO:tasks.rados.rados.0.smithi104.stdout:202: finishing write tid 3 to smithi10428818-37 2020-03-09T00:25:12.771 INFO:tasks.rados.rados.0.smithi104.stdout:202: finishing write tid 5 to smithi10428818-37 2020-03-09T00:25:12.771 INFO:tasks.rados.rados.0.smithi104.stdout:202: finishing write tid 6 to smithi10428818-37 2020-03-09T00:25:12.771 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 37 v 93 (ObjNum 96 snap 16 seq_num 96) dirty exists 2020-03-09T00:25:12.772 INFO:tasks.rados.rados.0.smithi104.stdout:202: left oid 37 (ObjNum 96 snap 16 seq_num 96) 2020-03-09T00:25:12.772 INFO:tasks.rados.rados.0.smithi104.stdout:202: done (2 left) 2020-03-09T00:25:12.772 INFO:tasks.rados.rados.0.smithi104.stdout:205: snap_remove snap 15 2020-03-09T00:25:12.779 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:12.775+0000 7f8b1181d700 -1 osd.14 pg_epoch: 63 pg[3.7s0( v 63'28 (0'0,63'28] local-lis/les=54/55 n=5 ec=32/32 lis/c=54/54 les/c/f=55/55/0 sis=54) [14,8,9,4,2,6,11]p14(0) r=0 lpr=54 luod=62'26 crt=62'26 lcod 55'24 mlcod 55'24 active+clean+snaptrim trimq=[5~1] ps=[f~3,14~1]] removing snap head 2020-03-09T00:25:12.787 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is unset 2020-03-09T00:25:12.848 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:12.847+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:12.850 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:12.850 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:12.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:12.850 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:12.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:25:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:25:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.009600803000001, 2020-03-09T00:25:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048492969999999998, 2020-03-09T00:25:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.851 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:25:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:25:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026746099999999998 2020-03-09T00:25:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837654+0000", 2020-03-09T00:25:12.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8997e-05 2020-03-09T00:25:12.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837683+0000", 2020-03-09T00:25:12.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4249999999999999e-05 2020-03-09T00:25:12.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837717+0000", 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9910000000000002e-06 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837725+0000", 2020-03-09T00:25:12.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013534699999999999 2020-03-09T00:25:12.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837861+0000", 2020-03-09T00:25:12.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9768e-05 2020-03-09T00:25:12.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837890+0000", 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042879290000000002 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842178+0000", 2020-03-09T00:25:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7553999999999998e-05 2020-03-09T00:25:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842236+0000", 2020-03-09T00:25:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048492969999999998 2020-03-09T00:25:12.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.864016+0000", 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.982971234000001, 2020-03-09T00:25:12.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044756739999999998, 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864016+0000", 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864016+0000", 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011600200000000001 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864132+0000", 2020-03-09T00:25:12.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8845999999999999e-05 2020-03-09T00:25:12.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864151+0000", 2020-03-09T00:25:12.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2444e-05 2020-03-09T00:25:12.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864164+0000", 2020-03-09T00:25:12.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.989e-06 2020-03-09T00:25:12.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864169+0000", 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7182999999999998e-05 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864226+0000", 2020-03-09T00:25:12.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.306e-05 2020-03-09T00:25:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.864239+0000", 2020-03-09T00:25:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042089520000000002 2020-03-09T00:25:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.868448+0000", 2020-03-09T00:25:12.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4197999999999999e-05 2020-03-09T00:25:12.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.868492+0000", 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044756739999999998 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs2 43/32 ECSubWrite(tid=27, reqid=client.4751.0:237, at_version=43'10, trim_to=0'0, roll_forward_to=39'8))", 2020-03-09T00:25:12.867 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:52.849011+0000", 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.997976691000002, 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042541499999999999, 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849011+0000", 2020-03-09T00:25:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849011+0000", 2020-03-09T00:25:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036331200000000002 2020-03-09T00:25:12.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849374+0000", 2020-03-09T00:25:12.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.58e-05 2020-03-09T00:25:12.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849400+0000", 2020-03-09T00:25:12.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7555e-05 2020-03-09T00:25:12.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849417+0000", 2020-03-09T00:25:12.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0299999999999994e-06 2020-03-09T00:25:12.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849425+0000", 2020-03-09T00:25:12.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5386e-05 2020-03-09T00:25:12.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849501+0000", 2020-03-09T00:25:12.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2916e-05 2020-03-09T00:25:12.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.849514+0000", 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036913200000000001 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.853205+0000", 2020-03-09T00:25:12.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9830999999999997e-05 2020-03-09T00:25:12.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:52.853265+0000", 2020-03-09T00:25:12.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042541499999999999 2020-03-09T00:25:12.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:12.878 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.996587443999999, 2020-03-09T00:25:12.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057419319999999999, 2020-03-09T00:25:12.879 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.879 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.879 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:12.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:12.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056030900000000005 2020-03-09T00:25:12.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850960+0000", 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9046e-05 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850979+0000", 2020-03-09T00:25:12.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0473e-05 2020-03-09T00:25:12.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850990+0000", 2020-03-09T00:25:12.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.198e-06 2020-03-09T00:25:12.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850995+0000", 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123569 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.851119+0000", 2020-03-09T00:25:12.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4375e-05 2020-03-09T00:25:12.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.851133+0000", 2020-03-09T00:25:12.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049526300000000004 2020-03-09T00:25:12.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856086+0000", 2020-03-09T00:25:12.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6332000000000002e-05 2020-03-09T00:25:12.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856142+0000", 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057419319999999999 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.886 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.982999484, 2020-03-09T00:25:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060463440000000004, 2020-03-09T00:25:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:12.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088386200000000002 2020-03-09T00:25:12.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864872+0000", 2020-03-09T00:25:12.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2838e-05 2020-03-09T00:25:12.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864905+0000", 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9593e-05 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864924+0000", 2020-03-09T00:25:12.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8414999999999999e-05 2020-03-09T00:25:12.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864943+0000", 2020-03-09T00:25:12.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000227028 2020-03-09T00:25:12.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865170+0000", 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7546000000000001e-05 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865187+0000", 2020-03-09T00:25:12.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047657489999999997 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.869953+0000", 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1312999999999999e-05 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.870034+0000", 2020-03-09T00:25:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060463440000000004 2020-03-09T00:25:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs2 44/32 ECSubWrite(tid=34, reqid=client.4751.0:249, at_version=44'16, trim_to=0'0, roll_forward_to=44'14))", 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.867324+0000", 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.979663643999999, 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045395000000000001, 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867324+0000", 2020-03-09T00:25:12.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867324+0000", 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035438100000000002 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867678+0000", 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7268999999999999e-05 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867705+0000", 2020-03-09T00:25:12.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0743000000000001e-05 2020-03-09T00:25:12.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867726+0000", 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6490000000000003e-06 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.867736+0000", 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033996200000000002 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868076+0000", 2020-03-09T00:25:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0281e-05 2020-03-09T00:25:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868086+0000", 2020-03-09T00:25:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037215719999999998 2020-03-09T00:25:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.871808+0000", 2020-03-09T00:25:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5643000000000002e-05 2020-03-09T00:25:12.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.871863+0000", 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045395000000000001 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.903 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/32 ECSubWrite(tid=22, reqid=client.4751.0:275, at_version=48'12, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:25:12.903 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:12.903 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.059061764000001, 2020-03-09T00:25:12.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080719139999999995, 2020-03-09T00:25:12.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.903 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:12.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013854899999999999 2020-03-09T00:25:12.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788064+0000", 2020-03-09T00:25:12.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9998000000000001e-05 2020-03-09T00:25:12.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788084+0000", 2020-03-09T00:25:12.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2672e-05 2020-03-09T00:25:12.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788097+0000", 2020-03-09T00:25:12.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5619999999999998e-06 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788104+0000", 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0817999999999999e-05 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788165+0000", 2020-03-09T00:25:12.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9372999999999999e-05 2020-03-09T00:25:12.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788185+0000", 2020-03-09T00:25:12.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077469949999999996 2020-03-09T00:25:12.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795932+0000", 2020-03-09T00:25:12.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5946999999999997e-05 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795998+0000", 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080719139999999995 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.911 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.912 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/32 ECSubWrite(tid=23, reqid=client.4751.0:276, at_version=48'13, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:25:12.912 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792573+0000", 2020-03-09T00:25:12.912 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.054414317000001, 2020-03-09T00:25:12.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043129509999999998, 2020-03-09T00:25:12.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792573+0000", 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792573+0000", 2020-03-09T00:25:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000185861 2020-03-09T00:25:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792759+0000", 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5899000000000002e-05 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792805+0000", 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2298000000000001e-05 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792837+0000", 2020-03-09T00:25:12.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1747e-05 2020-03-09T00:25:12.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792849+0000", 2020-03-09T00:25:12.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038443199999999998 2020-03-09T00:25:12.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.793233+0000", 2020-03-09T00:25:12.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8953e-05 2020-03-09T00:25:12.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.793252+0000", 2020-03-09T00:25:12.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003555192 2020-03-09T00:25:12.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796808+0000", 2020-03-09T00:25:12.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8568999999999999e-05 2020-03-09T00:25:12.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796886+0000", 2020-03-09T00:25:12.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043129509999999998 2020-03-09T00:25:12.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.920 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:12.921 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:12.921 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.029900681999999, 2020-03-09T00:25:12.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052768320000000004, 2020-03-09T00:25:12.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.921 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:12.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040852899999999998 2020-03-09T00:25:12.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817495+0000", 2020-03-09T00:25:12.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1301000000000002e-05 2020-03-09T00:25:12.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817517+0000", 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4061e-05 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817531+0000", 2020-03-09T00:25:12.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0720000000000001e-06 2020-03-09T00:25:12.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817535+0000", 2020-03-09T00:25:12.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012169500000000001 2020-03-09T00:25:12.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817656+0000", 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3953e-05 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817670+0000", 2020-03-09T00:25:12.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046400950000000003 2020-03-09T00:25:12.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822310+0000", 2020-03-09T00:25:12.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3125999999999999e-05 2020-03-09T00:25:12.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822364+0000", 2020-03-09T00:25:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052768320000000004 2020-03-09T00:25:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(3.1s3 53/52 [PushOp(3:85442c74:::smithi10428818-19:head, version: 49'20, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:85442c74:::smithi10428818-19:head@49'20, size: 1359872, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:25:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.888244+0000", 2020-03-09T00:25:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.958743685, 2020-03-09T00:25:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043367079999999999, 2020-03-09T00:25:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.929 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.888244+0000", 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.888244+0000", 2020-03-09T00:25:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026640799999999999 2020-03-09T00:25:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.888510+0000", 2020-03-09T00:25:12.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9816000000000001e-05 2020-03-09T00:25:12.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.888530+0000", 2020-03-09T00:25:12.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3519999999999998e-05 2020-03-09T00:25:12.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.888554+0000", 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0161e-05 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.888564+0000", 2020-03-09T00:25:12.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5320000000000004e-05 2020-03-09T00:25:12.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.888639+0000", 2020-03-09T00:25:12.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003941483 2020-03-09T00:25:12.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.892580+0000", 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043367079999999999 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.936 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(3.1s3 53/52 [PushOp(3:8a6d1775:::smithi10428818-12:11, version: 53'21, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:8a6d1775:::smithi10428818-12:11@53'21, size: 1261568, copy_subset: [], clone_subset: {}, snapset: 11=[]:{c=[c,5,4],11=[11,10,f]}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:25:12.936 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.906649+0000", 2020-03-09T00:25:12.936 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9403380590000001, 2020-03-09T00:25:12.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043491279999999998, 2020-03-09T00:25:12.936 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.936 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.936 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.906649+0000", 2020-03-09T00:25:12.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.906649+0000", 2020-03-09T00:25:12.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023671599999999999 2020-03-09T00:25:12.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.906886+0000", 2020-03-09T00:25:12.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4354e-05 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.906900+0000", 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1424999999999999e-05 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.906922+0000", 2020-03-09T00:25:12.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9280000000000004e-06 2020-03-09T00:25:12.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.906929+0000", 2020-03-09T00:25:12.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8194e-05 2020-03-09T00:25:12.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.906977+0000", 2020-03-09T00:25:12.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040215110000000002 2020-03-09T00:25:12.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.910999+0000", 2020-03-09T00:25:12.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043491279999999998 2020-03-09T00:25:12.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.942 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(3.1s3 53/52 [PushOp(3:8a6d1775:::smithi10428818-12:head, version: 53'24, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:8a6d1775:::smithi10428818-12:head@53'24, size: 2523136, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.917649+0000", 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9293388609999997, 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041737299999999996, 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.943 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.944 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.917649+0000", 2020-03-09T00:25:12.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.917649+0000", 2020-03-09T00:25:12.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039799200000000001 2020-03-09T00:25:12.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.918047+0000", 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.078e-05 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.918057+0000", 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5378000000000001e-05 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.918073+0000", 2020-03-09T00:25:12.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9360000000000002e-06 2020-03-09T00:25:12.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.918080+0000", 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1853000000000001e-05 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.918122+0000", 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037007910000000001 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.921822+0000", 2020-03-09T00:25:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041737299999999996 2020-03-09T00:25:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.950 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(3.1s3 53/52 [PushOp(3:8a6d1775:::smithi10428818-12:head, version: 53'24, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8a6d1775:::smithi10428818-12:head@53'24, size: 2523136, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2020-03-09T00:25:12.950 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.924833+0000", 2020-03-09T00:25:12.950 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9221544819999998, 2020-03-09T00:25:12.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040565729999999999, 2020-03-09T00:25:12.950 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.950 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.951 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.924833+0000", 2020-03-09T00:25:12.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.924833+0000", 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026814999999999999 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.925101+0000", 2020-03-09T00:25:12.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0159999999999999e-05 2020-03-09T00:25:12.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.925111+0000", 2020-03-09T00:25:12.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7790000000000001e-05 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.925129+0000", 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4080000000000001e-06 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.925136+0000", 2020-03-09T00:25:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4252999999999998e-05 2020-03-09T00:25:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.925181+0000", 2020-03-09T00:25:12.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037088120000000001 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.928890+0000", 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040565729999999999 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.957 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/32 ECSubWrite(tid=108, reqid=client.4751.0:368, at_version=54'44, trim_to=0'0, roll_forward_to=53'42))", 2020-03-09T00:25:12.957 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.795876+0000", 2020-03-09T00:25:12.957 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.0511110800000001, 2020-03-09T00:25:12.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047237169999999997, 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.795876+0000", 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.795876+0000", 2020-03-09T00:25:12.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035807200000000002 2020-03-09T00:25:12.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.796234+0000", 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3562000000000002e-05 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.796268+0000", 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8172000000000002e-05 2020-03-09T00:25:12.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.796286+0000", 2020-03-09T00:25:12.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5359999999999992e-06 2020-03-09T00:25:12.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.796295+0000", 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054681700000000001 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.796842+0000", 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8580000000000006e-06 2020-03-09T00:25:12.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.796850+0000", 2020-03-09T00:25:12.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037010720000000001 2020-03-09T00:25:12.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800551+0000", 2020-03-09T00:25:12.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8628000000000003e-05 2020-03-09T00:25:12.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800600+0000", 2020-03-09T00:25:12.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047237169999999997 2020-03-09T00:25:12.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.965 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.965 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.2s5 54/48 ECSubWrite(tid=7, reqid=client.4751.0:375, at_version=54'16, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:12.965 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.804082+0000", 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.0429058470000001, 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041751920000000003, 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804082+0000", 2020-03-09T00:25:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804082+0000", 2020-03-09T00:25:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117658 2020-03-09T00:25:12.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804199+0000", 2020-03-09T00:25:12.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3379000000000001e-05 2020-03-09T00:25:12.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804223+0000", 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2139e-05 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804235+0000", 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5130000000000004e-06 2020-03-09T00:25:12.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804240+0000", 2020-03-09T00:25:12.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6845999999999997e-05 2020-03-09T00:25:12.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804297+0000", 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2155e-05 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804319+0000", 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003895113 2020-03-09T00:25:12.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808214+0000", 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2388999999999997e-05 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808257+0000", 2020-03-09T00:25:12.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041751920000000003 2020-03-09T00:25:12.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.973 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 54/32 ECSubWrite(tid=77, reqid=client.4751.0:415, at_version=54'44, trim_to=0'0, roll_forward_to=54'42))", 2020-03-09T00:25:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.6440451990000007, 2020-03-09T00:25:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066307969999999999, 2020-03-09T00:25:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:12.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055621700000000002 2020-03-09T00:25:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203498+0000", 2020-03-09T00:25:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0248999999999999e-05 2020-03-09T00:25:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203519+0000", 2020-03-09T00:25:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3159000000000001e-05 2020-03-09T00:25:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203532+0000", 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8510000000000004e-06 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203538+0000", 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1504e-05 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203589+0000", 2020-03-09T00:25:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5809000000000001e-05 2020-03-09T00:25:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203605+0000", 2020-03-09T00:25:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059129430000000004 2020-03-09T00:25:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.209518+0000", 2020-03-09T00:25:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5065000000000002e-05 2020-03-09T00:25:12.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.209573+0000", 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066307969999999999 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s1 54/32 ECSubWrite(tid=114, reqid=client.4751.0:423, at_version=54'46, trim_to=0'0, roll_forward_to=54'44))", 2020-03-09T00:25:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.217294+0000", 2020-03-09T00:25:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.6296933550000006, 2020-03-09T00:25:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045773200000000002, 2020-03-09T00:25:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.982 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217294+0000", 2020-03-09T00:25:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217294+0000", 2020-03-09T00:25:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0002386 2020-03-09T00:25:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217533+0000", 2020-03-09T00:25:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6784000000000001e-05 2020-03-09T00:25:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217549+0000", 2020-03-09T00:25:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2417e-05 2020-03-09T00:25:12.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217562+0000", 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5780000000000004e-06 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217568+0000", 2020-03-09T00:25:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5118000000000003e-05 2020-03-09T00:25:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217624+0000", 2020-03-09T00:25:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.271e-05 2020-03-09T00:25:12.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.217636+0000", 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041912479999999998 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.221828+0000", 2020-03-09T00:25:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3865000000000001e-05 2020-03-09T00:25:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.221871+0000", 2020-03-09T00:25:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045773200000000002 2020-03-09T00:25:12.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s3 59/54 ECSubWrite(tid=164, reqid=client.4751.0:490, at_version=59'45, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.836818+0000", 2020-03-09T00:25:12.990 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0101694170000002, 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049356169999999998, 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.836818+0000", 2020-03-09T00:25:12.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.836818+0000", 2020-03-09T00:25:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059775900000000001 2020-03-09T00:25:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837416+0000", 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7926000000000001e-05 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837434+0000", 2020-03-09T00:25:12.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8789999999999992e-06 2020-03-09T00:25:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837443+0000", 2020-03-09T00:25:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4440000000000001e-06 2020-03-09T00:25:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837447+0000", 2020-03-09T00:25:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2846999999999998e-05 2020-03-09T00:25:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:12.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837490+0000", 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1929e-05 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837502+0000", 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042011940000000001 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.841703+0000", 2020-03-09T00:25:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0639000000000001e-05 2020-03-09T00:25:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:12.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.841754+0000", 2020-03-09T00:25:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049356169999999998 2020-03-09T00:25:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.998 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s3 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.843432+0000", 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.003555338, 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052060020000000004, 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:12.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843432+0000", 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843432+0000", 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011910670000000001 2020-03-09T00:25:13.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844623+0000", 2020-03-09T00:25:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5556999999999999e-05 2020-03-09T00:25:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844649+0000", 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9201000000000001e-05 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844668+0000", 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2920000000000001e-06 2020-03-09T00:25:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844674+0000", 2020-03-09T00:25:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4917000000000002e-05 2020-03-09T00:25:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844739+0000", 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6398000000000001e-05 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844756+0000", 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038280250000000001 2020-03-09T00:25:13.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.848584+0000", 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4545e-05 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:13.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.848638+0000", 2020-03-09T00:25:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052060020000000004 2020-03-09T00:25:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.9328793690000001, 2020-03-09T00:25:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094365569999999999, 2020-03-09T00:25:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.007 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011685599999999999 2020-03-09T00:25:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:13.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915277+0000", 2020-03-09T00:25:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3901999999999999e-05 2020-03-09T00:25:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915301+0000", 2020-03-09T00:25:13.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4554e-05 2020-03-09T00:25:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915315+0000", 2020-03-09T00:25:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8169999999999999e-06 2020-03-09T00:25:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915323+0000", 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2163999999999998e-05 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915365+0000", 2020-03-09T00:25:13.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4710000000000001e-06 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915373+0000", 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081172520000000001 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923490+0000", 2020-03-09T00:25:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4836999999999997e-05 2020-03-09T00:25:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923545+0000", 2020-03-09T00:25:13.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094365569999999999 2020-03-09T00:25:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:13.015 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:13.015 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:13.017 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:25:13.017 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:13.014+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:13.118 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:13.115+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:13.134 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:13.134 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:13.135 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:13.135 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:13.143 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:25:13.219 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:13.216+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:13.259 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:13.260 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:13.260 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:13.260 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:13.260 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:13.267 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:25:13.320 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:13.318+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:13.384 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:13.384 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:13.384 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:13.384 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:13.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.385 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:13.385 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:13.385 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.531610828000002, 2020-03-09T00:25:13.385 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999, 2020-03-09T00:25:13.386 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.386 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.386 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.386 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.386 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:13.387 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.387 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.387 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.387 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:13.388 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017622699999999999 2020-03-09T00:25:13.388 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.388 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.388 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850137+0000", 2020-03-09T00:25:13.388 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1974000000000003e-05 2020-03-09T00:25:13.389 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.389 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.389 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.389 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850169+0000", 2020-03-09T00:25:13.389 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4702000000000002e-05 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850193+0000", 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5790000000000004e-06 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.390 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.391 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850202+0000", 2020-03-09T00:25:13.391 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013641599999999999 2020-03-09T00:25:13.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.391 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.391 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.391 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850338+0000", 2020-03-09T00:25:13.392 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4691999999999999e-05 2020-03-09T00:25:13.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.392 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.392 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.392 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850353+0000", 2020-03-09T00:25:13.392 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011849303 2020-03-09T00:25:13.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.393 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.393 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862202+0000", 2020-03-09T00:25:13.393 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1013000000000001e-05 2020-03-09T00:25:13.393 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.394 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.394 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862253+0000", 2020-03-09T00:25:13.394 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999 2020-03-09T00:25:13.394 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.394 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.394 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.395 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:13.395 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:13.395 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.525415423999998, 2020-03-09T00:25:13.395 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595, 2020-03-09T00:25:13.395 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.396 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.396 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.396 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.396 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.396 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:13.396 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.397 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.397 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.397 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:13.397 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107948 2020-03-09T00:25:13.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.398 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.398 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.398 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856264+0000", 2020-03-09T00:25:13.398 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4422000000000001e-05 2020-03-09T00:25:13.398 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.398 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.398 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.399 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856288+0000", 2020-03-09T00:25:13.399 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5265e-05 2020-03-09T00:25:13.399 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.399 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.399 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.399 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856304+0000", 2020-03-09T00:25:13.400 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3209999999999994e-06 2020-03-09T00:25:13.400 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.400 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.400 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.400 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856313+0000", 2020-03-09T00:25:13.400 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00056587900000000003 2020-03-09T00:25:13.401 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.401 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.401 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.401 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856879+0000", 2020-03-09T00:25:13.401 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1976e-05 2020-03-09T00:25:13.401 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.402 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.402 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.402 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856891+0000", 2020-03-09T00:25:13.402 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098355160000000007 2020-03-09T00:25:13.402 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.403 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.403 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.403 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866726+0000", 2020-03-09T00:25:13.403 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6267999999999996e-05 2020-03-09T00:25:13.403 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.404 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.404 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.404 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866802+0000", 2020-03-09T00:25:13.404 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595 2020-03-09T00:25:13.404 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.404 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.404 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.405 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.405 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.405 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:13.405 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.864672+0000", 2020-03-09T00:25:13.405 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.516899582000001, 2020-03-09T00:25:13.405 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0080159729999999992, 2020-03-09T00:25:13.406 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.406 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.406 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.406 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.406 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864672+0000", 2020-03-09T00:25:13.407 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.407 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.407 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.407 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.407 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864672+0000", 2020-03-09T00:25:13.407 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018526400000000001 2020-03-09T00:25:13.408 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.408 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.408 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.408 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864857+0000", 2020-03-09T00:25:13.408 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8279000000000001e-05 2020-03-09T00:25:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864885+0000", 2020-03-09T00:25:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6443000000000002e-05 2020-03-09T00:25:13.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.410 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.410 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.410 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864902+0000", 2020-03-09T00:25:13.410 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6979999999999998e-06 2020-03-09T00:25:13.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.411 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.411 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.411 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864909+0000", 2020-03-09T00:25:13.411 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9886999999999998e-05 2020-03-09T00:25:13.411 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.411 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.412 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.412 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864969+0000", 2020-03-09T00:25:13.412 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4875000000000001e-05 2020-03-09T00:25:13.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.413 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.413 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.413 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864984+0000", 2020-03-09T00:25:13.413 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0076501599999999996 2020-03-09T00:25:13.413 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.413 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.414 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.414 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.872634+0000", 2020-03-09T00:25:13.414 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3366999999999999e-05 2020-03-09T00:25:13.414 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.414 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.415 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.415 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.872688+0000", 2020-03-09T00:25:13.415 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0080159729999999992 2020-03-09T00:25:13.415 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.415 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.415 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.416 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.416 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.416 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 47/46 ECSubWrite(tid=26, reqid=client.4751.0:265, at_version=47'10, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:13.416 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:56.782000+0000", 2020-03-09T00:25:13.416 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.599571034, 2020-03-09T00:25:13.416 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088401729999999998, 2020-03-09T00:25:13.416 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.417 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.417 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.417 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.417 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.417 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782000+0000", 2020-03-09T00:25:13.417 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.418 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.418 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.418 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.418 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782000+0000", 2020-03-09T00:25:13.418 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00075502399999999997 2020-03-09T00:25:13.418 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.419 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.419 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.419 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782755+0000", 2020-03-09T00:25:13.419 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5735999999999999e-05 2020-03-09T00:25:13.419 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.419 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.420 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.420 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782781+0000", 2020-03-09T00:25:13.420 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6680999999999998e-05 2020-03-09T00:25:13.420 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.420 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.420 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782798+0000", 2020-03-09T00:25:13.421 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6900000000000003e-06 2020-03-09T00:25:13.421 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.421 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.421 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.422 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.782804+0000", 2020-03-09T00:25:13.422 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00042437399999999999 2020-03-09T00:25:13.422 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.423 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.423 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.423 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783229+0000", 2020-03-09T00:25:13.423 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4287000000000001e-05 2020-03-09T00:25:13.423 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.424 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.424 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.783243+0000", 2020-03-09T00:25:13.424 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066180969999999999 2020-03-09T00:25:13.424 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.789861+0000", 2020-03-09T00:25:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00097928399999999997 2020-03-09T00:25:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:56.790840+0000", 2020-03-09T00:25:13.426 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088401729999999998 2020-03-09T00:25:13.426 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.426 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.426 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.426 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(osd.10.23:4 3.3s0 3:c607ec57:::smithi10428818-30:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e48)", 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.592442202999999, 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999, 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:13.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.428 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.428 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:13.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8078999999999999e-05 2020-03-09T00:25:13.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.429 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.429 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789167+0000", 2020-03-09T00:25:13.429 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1499999999999998e-06 2020-03-09T00:25:13.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.430 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.430 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789172+0000", 2020-03-09T00:25:13.430 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6209999999999993e-06 2020-03-09T00:25:13.430 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.430 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.430 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.431 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789181+0000", 2020-03-09T00:25:13.431 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4939999999999997e-06 2020-03-09T00:25:13.431 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.431 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.431 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.431 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789185+0000", 2020-03-09T00:25:13.431 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5793e-05 2020-03-09T00:25:13.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.432 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789231+0000", 2020-03-09T00:25:13.432 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7570000000000002e-06 2020-03-09T00:25:13.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.433 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.433 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:13.433 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789235+0000", 2020-03-09T00:25:13.433 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.99423119699999996 2020-03-09T00:25:13.433 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783466+0000", 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000227851 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:13.434 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783694+0000", 2020-03-09T00:25:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056076839999999999 2020-03-09T00:25:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789302+0000", 2020-03-09T00:25:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999 2020-03-09T00:25:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.436 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.436 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.436 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.436 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.436 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 53/52 ECSubWriteReply(tid=1, last_complete=34'2, committed=1, applied=1))", 2020-03-09T00:25:13.436 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:13.436 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.588273069, 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215, 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:13.437 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.438 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.438 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:13.438 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9154999999999999e-05 2020-03-09T00:25:13.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793357+0000", 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4100000000000008e-06 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793366+0000", 2020-03-09T00:25:13.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3783e-05 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793380+0000", 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6469999999999998e-06 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.440 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793387+0000", 2020-03-09T00:25:13.441 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012002081 2020-03-09T00:25:13.441 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.441 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805389+0000", 2020-03-09T00:25:13.441 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0139000000000001e-05 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805409+0000", 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.442 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 53/52 ECSubWriteReply(tid=1, last_complete=53'15, committed=1, applied=1))", 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.797636+0000", 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.583935718999999, 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078236859999999998, 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.443 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.444 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.444 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.444 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.444 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.797636+0000", 2020-03-09T00:25:13.444 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.444 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.444 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.445 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.445 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.797636+0000", 2020-03-09T00:25:13.445 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6745e-05 2020-03-09T00:25:13.445 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.445 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.445 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.445 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.797692+0000", 2020-03-09T00:25:13.446 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9820000000000003e-06 2020-03-09T00:25:13.446 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.446 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.446 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.446 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.797702+0000", 2020-03-09T00:25:13.446 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6572999999999999e-05 2020-03-09T00:25:13.446 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.797719+0000", 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.396e-06 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.447 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.797726+0000", 2020-03-09T00:25:13.448 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0077154980000000003 2020-03-09T00:25:13.448 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.448 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.448 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.448 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805442+0000", 2020-03-09T00:25:13.448 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7492000000000001e-05 2020-03-09T00:25:13.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.449 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.449 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.449 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805459+0000", 2020-03-09T00:25:13.449 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078236859999999998 2020-03-09T00:25:13.449 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.449 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 53/52 ECSubWriteReply(tid=1, last_complete=53'15, committed=1, applied=1))", 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.798247+0000", 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.583324595000001, 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072603010000000003, 2020-03-09T00:25:13.450 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.451 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.451 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.451 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.451 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.451 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.798247+0000", 2020-03-09T00:25:13.451 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.452 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.798247+0000", 2020-03-09T00:25:13.452 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9299999999999998e-05 2020-03-09T00:25:13.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.798306+0000", 2020-03-09T00:25:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7279999999999995e-06 2020-03-09T00:25:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.798314+0000", 2020-03-09T00:25:13.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4107000000000001e-05 2020-03-09T00:25:13.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.454 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.454 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.798328+0000", 2020-03-09T00:25:13.454 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.626e-06 2020-03-09T00:25:13.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.798333+0000", 2020-03-09T00:25:13.455 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071559520000000001 2020-03-09T00:25:13.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805489+0000", 2020-03-09T00:25:13.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7588000000000002e-05 2020-03-09T00:25:13.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.456 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.456 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.456 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805507+0000", 2020-03-09T00:25:13.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0072603010000000003 2020-03-09T00:25:13.456 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 53/52 ECSubWrite(tid=47, reqid=client.4751.0:355, at_version=53'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.530388325000001, 2020-03-09T00:25:13.457 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252, 2020-03-09T00:25:13.458 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.458 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.458 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.458 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.458 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:13.458 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.459 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.459 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:13.459 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00081411500000000002 2020-03-09T00:25:13.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.460 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.460 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851997+0000", 2020-03-09T00:25:13.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0162999999999999e-05 2020-03-09T00:25:13.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852027+0000", 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7360999999999999e-05 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852045+0000", 2020-03-09T00:25:13.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2069000000000001e-05 2020-03-09T00:25:13.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.462 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.462 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852057+0000", 2020-03-09T00:25:13.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000131192 2020-03-09T00:25:13.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852188+0000", 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.517e-05 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852203+0000", 2020-03-09T00:25:13.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099256910000000004 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862129+0000", 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6758999999999997e-05 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862176+0000", 2020-03-09T00:25:13.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252 2020-03-09T00:25:13.465 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.465 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.465 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.466 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=159, reqid=osd.12.0:159, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:13.466 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:13.466 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5612569680000004, 2020-03-09T00:25:13.466 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001, 2020-03-09T00:25:13.466 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.466 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.467 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:13.467 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.468 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.468 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:13.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1613999999999997e-05 2020-03-09T00:25:13.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.468 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820376+0000", 2020-03-09T00:25:13.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6792e-05 2020-03-09T00:25:13.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.469 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.469 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820403+0000", 2020-03-09T00:25:13.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4728e-05 2020-03-09T00:25:13.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.470 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.470 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820417+0000", 2020-03-09T00:25:13.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7910000000000005e-06 2020-03-09T00:25:13.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820425+0000", 2020-03-09T00:25:13.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010933589999999999 2020-03-09T00:25:13.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821519+0000", 2020-03-09T00:25:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3885e-05 2020-03-09T00:25:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821533+0000", 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011394939999999999 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832927+0000", 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6371e-05 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832974+0000", 2020-03-09T00:25:13.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001 2020-03-09T00:25:13.474 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.474 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.474 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.475 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=5, reqid=client.4751.0:485, at_version=59'26, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:13.475 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:13.475 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5606150320000003, 2020-03-09T00:25:13.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083, 2020-03-09T00:25:13.475 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.475 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.476 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:13.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.477 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.477 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:13.477 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4487000000000002e-05 2020-03-09T00:25:13.477 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.477 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.478 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821011+0000", 2020-03-09T00:25:13.478 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9616e-05 2020-03-09T00:25:13.478 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.478 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.478 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.478 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821030+0000", 2020-03-09T00:25:13.478 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4766000000000001e-05 2020-03-09T00:25:13.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.479 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.479 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.479 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821045+0000", 2020-03-09T00:25:13.479 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.789e-06 2020-03-09T00:25:13.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.479 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.480 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.480 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821051+0000", 2020-03-09T00:25:13.480 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010220629999999999 2020-03-09T00:25:13.480 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.480 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.480 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.480 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822073+0000", 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1511999999999999e-05 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822085+0000", 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010894232 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832979+0000", 2020-03-09T00:25:13.482 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:13.482 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.482 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833022+0000", 2020-03-09T00:25:13.483 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083 2020-03-09T00:25:13.483 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.483 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.483 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.483 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=6, reqid=client.4751.0:486, at_version=59'27, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5604701499999996, 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978, 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.484 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:13.485 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4271000000000003e-05 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821155+0000", 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8627e-05 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.486 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.487 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821174+0000", 2020-03-09T00:25:13.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4825e-05 2020-03-09T00:25:13.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.487 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.487 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.487 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821189+0000", 2020-03-09T00:25:13.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4070000000000001e-06 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821194+0000", 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039590839999999999 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.488 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.489 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825153+0000", 2020-03-09T00:25:13.489 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3074999999999999e-05 2020-03-09T00:25:13.489 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.489 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.489 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.489 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825166+0000", 2020-03-09T00:25:13.489 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078593580000000003 2020-03-09T00:25:13.490 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.490 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.490 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.490 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833026+0000", 2020-03-09T00:25:13.490 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2330999999999999e-05 2020-03-09T00:25:13.490 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.490 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.491 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.491 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833058+0000", 2020-03-09T00:25:13.491 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978 2020-03-09T00:25:13.491 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.491 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.491 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.492 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.493 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.493 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=59'43, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:13.493 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:13.493 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5602651280000002, 2020-03-09T00:25:13.493 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525, 2020-03-09T00:25:13.493 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.494 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.494 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.494 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.494 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.495 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:13.495 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.495 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.495 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.495 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.495 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:13.495 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6616999999999997e-05 2020-03-09T00:25:13.496 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.496 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.496 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.496 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821363+0000", 2020-03-09T00:25:13.496 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1061e-05 2020-03-09T00:25:13.496 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.496 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821384+0000", 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5315000000000002e-05 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821399+0000", 2020-03-09T00:25:13.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0929999999999999e-06 2020-03-09T00:25:13.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821405+0000", 2020-03-09T00:25:13.498 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003024559 2020-03-09T00:25:13.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824430+0000", 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2744e-05 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824443+0000", 2020-03-09T00:25:13.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010025054 2020-03-09T00:25:13.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834468+0000", 2020-03-09T00:25:13.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0082e-05 2020-03-09T00:25:13.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834508+0000", 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.502 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=7, reqid=client.4751.0:487, at_version=59'28, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:13.502 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:13.502 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5419225369999996, 2020-03-09T00:25:13.502 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916, 2020-03-09T00:25:13.502 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.502 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.502 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:13.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7596999999999998e-05 2020-03-09T00:25:13.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839706+0000", 2020-03-09T00:25:13.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5558000000000002e-05 2020-03-09T00:25:13.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839732+0000", 2020-03-09T00:25:13.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0251999999999999e-05 2020-03-09T00:25:13.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839752+0000", 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0780000000000002e-06 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.506 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839761+0000", 2020-03-09T00:25:13.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010019800000000001 2020-03-09T00:25:13.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.507 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.507 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.507 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839861+0000", 2020-03-09T00:25:13.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9032e-05 2020-03-09T00:25:13.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.508 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.508 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839881+0000", 2020-03-09T00:25:13.508 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014066339000000001 2020-03-09T00:25:13.508 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.508 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853947+0000", 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000210862 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854158+0000", 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916 2020-03-09T00:25:13.509 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=164, reqid=client.4751.0:490, at_version=59'45, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5417041060000001, 2020-03-09T00:25:13.510 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001, 2020-03-09T00:25:13.511 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.511 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.511 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:13.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000195997 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.513 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840063+0000", 2020-03-09T00:25:13.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7005e-05 2020-03-09T00:25:13.513 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.513 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.513 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.513 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840090+0000", 2020-03-09T00:25:13.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6943999999999999e-05 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840107+0000", 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7489999999999998e-06 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.514 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840115+0000", 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019725999999999999 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840312+0000", 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4409e-05 2020-03-09T00:25:13.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840327+0000", 2020-03-09T00:25:13.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018701885000000001 2020-03-09T00:25:13.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859028+0000", 2020-03-09T00:25:13.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7057e-05 2020-03-09T00:25:13.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859076+0000", 2020-03-09T00:25:13.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001 2020-03-09T00:25:13.518 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.518 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.518 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.518 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:13.518 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5381953260000003, 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002, 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:13.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012568360000000001 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844633+0000", 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0247999999999999e-05 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844663+0000", 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6184e-05 2020-03-09T00:25:13.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844679+0000", 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8560000000000001e-06 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844686+0000", 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8074e-05 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844744+0000", 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9119000000000001e-05 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.524 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844763+0000", 2020-03-09T00:25:13.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014486196999999999 2020-03-09T00:25:13.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859250+0000", 2020-03-09T00:25:13.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115911 2020-03-09T00:25:13.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859365+0000", 2020-03-09T00:25:13.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002 2020-03-09T00:25:13.526 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.526 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.526 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.527 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=90, reqid=client.4751.0:499, at_version=59'56, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:13.527 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:13.527 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5324895539999996, 2020-03-09T00:25:13.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999, 2020-03-09T00:25:13.527 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:13.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050415699999999998 2020-03-09T00:25:13.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849586+0000", 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9476e-05 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849615+0000", 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8382000000000001e-05 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.530 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849634+0000", 2020-03-09T00:25:13.531 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0439999999999996e-06 2020-03-09T00:25:13.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849642+0000", 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018423140000000001 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851484+0000", 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3304e-05 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851497+0000", 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097988120000000005 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861296+0000", 2020-03-09T00:25:13.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1790999999999999e-05 2020-03-09T00:25:13.534 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861348+0000", 2020-03-09T00:25:13.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999 2020-03-09T00:25:13.534 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.534 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=91, reqid=client.4751.0:500, at_version=59'57, trim_to=0'0, roll_forward_to=59'53))", 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.853560+0000", 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.5280110970000003, 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078349349999999995, 2020-03-09T00:25:13.535 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853560+0000", 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853560+0000", 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107778 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853668+0000", 2020-03-09T00:25:13.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9055e-05 2020-03-09T00:25:13.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853687+0000", 2020-03-09T00:25:13.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5636000000000001e-05 2020-03-09T00:25:13.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853703+0000", 2020-03-09T00:25:13.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6510000000000004e-06 2020-03-09T00:25:13.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853709+0000", 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025953840000000001 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.856305+0000", 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2748000000000001e-05 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.856317+0000", 2020-03-09T00:25:13.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0050421290000000002 2020-03-09T00:25:13.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861360+0000", 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.5553999999999999e-05 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861395+0000", 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078349349999999995 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.542 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.543 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.543 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 63/54 ECSubWrite(tid=100, reqid=osd.6.0:100, at_version=63'66, trim_to=0'0, roll_forward_to=63'64))", 2020-03-09T00:25:13.543 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.785518+0000", 2020-03-09T00:25:13.543 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.59605336399999997, 2020-03-09T00:25:13.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0085199899999999999, 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.785518+0000", 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.785518+0000", 2020-03-09T00:25:13.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9917000000000002e-05 2020-03-09T00:25:13.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.785578+0000", 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8804999999999999e-05 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.785607+0000", 2020-03-09T00:25:13.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3692e-05 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.785620+0000", 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2719999999999997e-06 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.785626+0000", 2020-03-09T00:25:13.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00098861900000000004 2020-03-09T00:25:13.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786614+0000", 2020-03-09T00:25:13.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2327000000000001e-05 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786627+0000", 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073721539999999997 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.793999+0000", 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9203999999999999e-05 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.794038+0000", 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0085199899999999999 2020-03-09T00:25:13.550 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 63/32 ECSubWrite(tid=114, reqid=osd.8.0:114, at_version=63'79, trim_to=0'0, roll_forward_to=60'71))", 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.786740+0000", 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.59483135700000001, 2020-03-09T00:25:13.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073543970000000004, 2020-03-09T00:25:13.552 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:13.552 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.552 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:13.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:13.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786740+0000", 2020-03-09T00:25:13.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786740+0000", 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7893999999999998e-05 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:13.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786798+0000", 2020-03-09T00:25:13.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0665e-05 2020-03-09T00:25:13.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:13.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786819+0000", 2020-03-09T00:25:13.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5121e-05 2020-03-09T00:25:13.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:13.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786834+0000", 2020-03-09T00:25:13.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1310000000000003e-06 2020-03-09T00:25:13.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786840+0000", 2020-03-09T00:25:13.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7599000000000003e-05 2020-03-09T00:25:13.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:13.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786887+0000", 2020-03-09T00:25:13.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2398000000000001e-05 2020-03-09T00:25:13.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.786900+0000", 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071520799999999999 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.794052+0000", 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2509000000000002e-05 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:13.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.794094+0000", 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073543970000000004 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:13.560 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:13.561 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:25:13.562 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:13.559+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:13.661 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:13.659+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:13.689 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:13.689 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:13.689 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:13.689 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:13.696 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:25:13.762 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:13.761+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:13.772 INFO:tasks.rados.rados.0.smithi104.stdout:205: done (2 left) 2020-03-09T00:25:13.772 INFO:tasks.rados.rados.0.smithi104.stdout:206: rollback oid 31 current snap is 16 2020-03-09T00:25:13.772 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 31 to 11 2020-03-09T00:25:13.772 INFO:tasks.rados.rados.0.smithi104.stdout:201: finishing copy_from to smithi10428818-23 2020-03-09T00:25:13.772 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 23 v 47 (ObjNum 82 snap 12 seq_num 82) dirty exists 2020-03-09T00:25:13.773 INFO:tasks.rados.rados.0.smithi104.stdout:201: finishing copy_from racing read to smithi10428818-23 2020-03-09T00:25:13.773 INFO:tasks.rados.rados.0.smithi104.stdout:204: finishing rollback tid 0 to smithi10428818-13 2020-03-09T00:25:13.773 INFO:tasks.rados.rados.0.smithi104.stdout:204: finishing rollback tid 1 to smithi10428818-13 2020-03-09T00:25:13.773 INFO:tasks.rados.rados.0.smithi104.stdout:204: finishing rollback tid 2 to smithi10428818-13 2020-03-09T00:25:13.773 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 13 v 75 (ObjNum 85 snap 13 seq_num 85) dirty exists 2020-03-09T00:25:13.773 INFO:tasks.rados.rados.0.smithi104.stdout:201: done (2 left) 2020-03-09T00:25:13.773 INFO:tasks.rados.rados.0.smithi104.stdout:204: done (1 left) 2020-03-09T00:25:13.774 INFO:tasks.rados.rados.0.smithi104.stdout:207: rollback oid 9 current snap is 16 2020-03-09T00:25:13.774 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 9 to 13 2020-03-09T00:25:13.774 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 7 current snap is 16 2020-03-09T00:25:13.774 INFO:tasks.rados.rados.0.smithi104.stdout:208: seq_num 97 ranges {81920=475136,557056=442368,999424=442368,1441792=278528} 2020-03-09T00:25:13.775 INFO:tasks.rados.rados.0.smithi104.stdout:208: writing smithi10428818-7 from 81920 to 557056 tid 1 2020-03-09T00:25:13.778 INFO:tasks.rados.rados.0.smithi104.stdout:208: writing smithi10428818-7 from 557056 to 999424 tid 2 2020-03-09T00:25:13.781 INFO:tasks.rados.rados.0.smithi104.stdout:208: writing smithi10428818-7 from 999424 to 1441792 tid 3 2020-03-09T00:25:13.784 INFO:tasks.rados.rados.0.smithi104.stdout:208: writing smithi10428818-7 from 1441792 to 1720320 tid 4 2020-03-09T00:25:13.784 INFO:tasks.rados.rados.0.smithi104.stdout:209: copy_from oid 5 from oid 26 current snap is 16 2020-03-09T00:25:13.784 INFO:tasks.rados.rados.0.smithi104.stdout:207: finishing rollback tid 1 to smithi10428818-9 2020-03-09T00:25:13.784 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 9 v 0 (ObjNum -326269968 snap 22099 seq_num 3967794688) dirty dne 2020-03-09T00:25:13.784 INFO:tasks.rados.rados.0.smithi104.stdout:206: finishing rollback tid 0 to smithi10428818-31 2020-03-09T00:25:13.784 INFO:tasks.rados.rados.0.smithi104.stdout:206: finishing rollback tid 1 to smithi10428818-31 2020-03-09T00:25:13.785 INFO:tasks.rados.rados.0.smithi104.stdout:207: done (3 left) 2020-03-09T00:25:13.785 INFO:tasks.rados.rados.0.smithi104.stdout:210: rollback oid 32 current snap is 16 2020-03-09T00:25:13.785 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 32 to 7 2020-03-09T00:25:13.785 INFO:tasks.rados.rados.0.smithi104.stdout:211: rollback oid 37 current snap is 16 2020-03-09T00:25:13.785 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 37 to 7 2020-03-09T00:25:13.785 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 15 current snap is 16 2020-03-09T00:25:13.785 INFO:tasks.rados.rados.0.smithi104.stdout:212: seq_num 98 ranges {229376=491520,720896=589824,1310720=557056,1867776=16384} 2020-03-09T00:25:13.788 INFO:tasks.rados.rados.0.smithi104.stdout:212: writing smithi10428818-15 from 229376 to 720896 tid 1 2020-03-09T00:25:13.792 INFO:tasks.rados.rados.0.smithi104.stdout:212: writing smithi10428818-15 from 720896 to 1310720 tid 2 2020-03-09T00:25:13.796 INFO:tasks.rados.rados.0.smithi104.stdout:212: writing smithi10428818-15 from 1310720 to 1867776 tid 3 2020-03-09T00:25:13.796 INFO:tasks.rados.rados.0.smithi104.stdout:212: writing smithi10428818-15 from 1867776 to 1884160 tid 4 2020-03-09T00:25:13.796 INFO:tasks.rados.rados.0.smithi104.stdout:213: snap_remove snap 13 2020-03-09T00:25:13.824 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:13.824 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:13.824 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:13.824 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:13.825 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:13.833 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:25:13.863 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:13.861+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:13.949 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:13.950 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:13.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:13.950 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:13.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.950 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:207 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 5] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:13.950 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:13.951 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.090027639999999, 2020-03-09T00:25:13.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001, 2020-03-09T00:25:13.951 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:13.951 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:13.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:13.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:13.952 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:13.952 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207 2020-03-09T00:25:13.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.952 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:13.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:13.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:13.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:13.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:13.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:13.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7176999999999996e-05 2020-03-09T00:25:13.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:13.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856995+0000", 2020-03-09T00:25:13.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7161999999999999e-05 2020-03-09T00:25:13.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:13.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857013+0000", 2020-03-09T00:25:13.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2879e-05 2020-03-09T00:25:13.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:13.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857026+0000", 2020-03-09T00:25:13.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6669999999999996e-06 2020-03-09T00:25:13.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857033+0000", 2020-03-09T00:25:13.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6767999999999997e-05 2020-03-09T00:25:13.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:13.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857080+0000", 2020-03-09T00:25:13.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4165999999999994e-05 2020-03-09T00:25:13.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:13.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857154+0000", 2020-03-09T00:25:13.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068311889999999997 2020-03-09T00:25:13.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.863985+0000", 2020-03-09T00:25:13.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041278249999999999 2020-03-09T00:25:13.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.960 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868113+0000", 2020-03-09T00:25:13.960 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001971132 2020-03-09T00:25:13.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.960 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:13.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870084+0000", 2020-03-09T00:25:13.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113266 2020-03-09T00:25:13.961 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.961 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.961 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:13.961 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870198+0000", 2020-03-09T00:25:13.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001 2020-03-09T00:25:13.962 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.962 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:13.962 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.962 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:208 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:13.963 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:13.963 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.089857894000001, 2020-03-09T00:25:13.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001, 2020-03-09T00:25:13.963 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:13.963 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:13.963 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:13.964 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:13.964 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:13.964 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 208 2020-03-09T00:25:13.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.964 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:13.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:13.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:13.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:13.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:13.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:13.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5717000000000003e-05 2020-03-09T00:25:13.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:13.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857134+0000", 2020-03-09T00:25:13.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3659999999999997e-06 2020-03-09T00:25:13.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:13.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857140+0000", 2020-03-09T00:25:13.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2546e-05 2020-03-09T00:25:13.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:13.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857153+0000", 2020-03-09T00:25:13.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7190000000000001e-06 2020-03-09T00:25:13.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857157+0000", 2020-03-09T00:25:13.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134509 2020-03-09T00:25:13.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:13.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857292+0000", 2020-03-09T00:25:13.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5519999999999999e-05 2020-03-09T00:25:13.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.971 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:13.971 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857327+0000", 2020-03-09T00:25:13.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010666132 2020-03-09T00:25:13.972 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.972 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.972 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:13.972 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867994+0000", 2020-03-09T00:25:13.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110682 2020-03-09T00:25:13.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.973 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:13.973 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868104+0000", 2020-03-09T00:25:13.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00321515 2020-03-09T00:25:13.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:13.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871319+0000", 2020-03-09T00:25:13.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011457 2020-03-09T00:25:13.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:13.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871434+0000", 2020-03-09T00:25:13.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013121699999999999 2020-03-09T00:25:13.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:13.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871565+0000", 2020-03-09T00:25:13.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001 2020-03-09T00:25:13.977 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.977 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:13.977 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.978 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 44/32 ECSubWrite(tid=44, reqid=osd.13.0:44, at_version=44'11, trim_to=0'0, roll_forward_to=42'9))", 2020-03-09T00:25:13.978 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:13.978 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.097804663000002, 2020-03-09T00:25:13.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002, 2020-03-09T00:25:13.978 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:13.979 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.979 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:13.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:13.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:13.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:13.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.980 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:13.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:13.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8740999999999999e-05 2020-03-09T00:25:13.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:13.981 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849200+0000", 2020-03-09T00:25:13.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2819e-05 2020-03-09T00:25:13.981 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.981 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.981 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:13.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849223+0000", 2020-03-09T00:25:13.982 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5366000000000002e-05 2020-03-09T00:25:13.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.982 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:13.982 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849238+0000", 2020-03-09T00:25:13.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7410000000000003e-06 2020-03-09T00:25:13.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849245+0000", 2020-03-09T00:25:13.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5445999999999996e-05 2020-03-09T00:25:13.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:13.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849310+0000", 2020-03-09T00:25:13.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3651e-05 2020-03-09T00:25:13.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849324+0000", 2020-03-09T00:25:13.985 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057096769999999998 2020-03-09T00:25:13.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.985 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.985 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855034+0000", 2020-03-09T00:25:13.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001136 2020-03-09T00:25:13.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:13.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855147+0000", 2020-03-09T00:25:13.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002 2020-03-09T00:25:13.986 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.987 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:13.987 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.987 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:13.987 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:13.988 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.096663077999999, 2020-03-09T00:25:13.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998, 2020-03-09T00:25:13.988 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:13.988 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.988 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:13.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:13.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:13.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:13.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:13.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:13.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018406799999999999 2020-03-09T00:25:13.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.990 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.990 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:13.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850467+0000", 2020-03-09T00:25:13.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2490999999999999e-05 2020-03-09T00:25:13.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.992 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:13.992 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850489+0000", 2020-03-09T00:25:13.992 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6807999999999999e-05 2020-03-09T00:25:13.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.992 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:13.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:13.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9419999999999995e-06 2020-03-09T00:25:13.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.993 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:13.993 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850514+0000", 2020-03-09T00:25:13.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088484100000000003 2020-03-09T00:25:13.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.994 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.994 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:13.994 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851399+0000", 2020-03-09T00:25:13.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2279999999999999e-05 2020-03-09T00:25:13.994 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:13.995 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851411+0000", 2020-03-09T00:25:13.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083268249999999995 2020-03-09T00:25:13.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:13.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859738+0000", 2020-03-09T00:25:13.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2222999999999998e-05 2020-03-09T00:25:13.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:13.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859790+0000", 2020-03-09T00:25:13.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998 2020-03-09T00:25:13.997 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.997 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:13.997 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:13.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:13.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.998 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:13.998 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:13.998 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.12954839, 2020-03-09T00:25:13.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001, 2020-03-09T00:25:13.998 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:13.999 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:13.999 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:13.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:13.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:13.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:13.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:13.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.000 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.000 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:14.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069548899999999998 2020-03-09T00:25:14.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818093+0000", 2020-03-09T00:25:14.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6591999999999999e-05 2020-03-09T00:25:14.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818120+0000", 2020-03-09T00:25:14.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5192e-05 2020-03-09T00:25:14.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818135+0000", 2020-03-09T00:25:14.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4989999999999997e-06 2020-03-09T00:25:14.003 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818142+0000", 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4997000000000002e-05 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818197+0000", 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3164e-05 2020-03-09T00:25:14.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.005 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.005 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818210+0000", 2020-03-09T00:25:14.005 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092824440000000008 2020-03-09T00:25:14.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.015 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.015 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827493+0000", 2020-03-09T00:25:14.015 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2645999999999997e-05 2020-03-09T00:25:14.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.016 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827546+0000", 2020-03-09T00:25:14.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001 2020-03-09T00:25:14.018 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.018 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.019 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.019 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:14.020 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.792363+0000", 2020-03-09T00:25:14.020 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.154583034, 2020-03-09T00:25:14.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050713349999999997, 2020-03-09T00:25:14.020 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.020 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.020 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792363+0000", 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792363+0000", 2020-03-09T00:25:14.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00059428200000000001 2020-03-09T00:25:14.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792957+0000", 2020-03-09T00:25:14.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3232000000000003e-05 2020-03-09T00:25:14.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792990+0000", 2020-03-09T00:25:14.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1162000000000002e-05 2020-03-09T00:25:14.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793012+0000", 2020-03-09T00:25:14.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5129999999999996e-06 2020-03-09T00:25:14.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793020+0000", 2020-03-09T00:25:14.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7120999999999997e-05 2020-03-09T00:25:14.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793077+0000", 2020-03-09T00:25:14.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8972999999999999e-05 2020-03-09T00:25:14.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793096+0000", 2020-03-09T00:25:14.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042877590000000004 2020-03-09T00:25:14.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.026 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.797384+0000", 2020-03-09T00:25:14.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0293000000000001e-05 2020-03-09T00:25:14.027 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.027 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.027 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.797434+0000", 2020-03-09T00:25:14.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050713349999999997 2020-03-09T00:25:14.028 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.028 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.028 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.028 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.028 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.028 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/52 ECSubWrite(tid=2, reqid=client.4751.0:328, at_version=53'17, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:14.028 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.801759+0000", 2020-03-09T00:25:14.029 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.145186636, 2020-03-09T00:25:14.029 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044983990000000001, 2020-03-09T00:25:14.029 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.029 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.029 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.029 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.029 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.801759+0000", 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.801759+0000", 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029739399999999998 2020-03-09T00:25:14.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802057+0000", 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0837999999999999e-05 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802078+0000", 2020-03-09T00:25:14.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1408e-05 2020-03-09T00:25:14.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802089+0000", 2020-03-09T00:25:14.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1730000000000003e-06 2020-03-09T00:25:14.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802094+0000", 2020-03-09T00:25:14.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9200999999999999e-05 2020-03-09T00:25:14.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802143+0000", 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3560000000000001e-05 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.802157+0000", 2020-03-09T00:25:14.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.004064837 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.806222+0000", 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5988000000000002e-05 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.806258+0000", 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044983990000000001 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.036 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/52 ECSubWrite(tid=3, reqid=client.4751.0:328, at_version=0'0, trim_to=0'0, roll_forward_to=53'17))", 2020-03-09T00:25:14.037 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.809490+0000", 2020-03-09T00:25:14.037 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.137456374999999, 2020-03-09T00:25:14.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054575800000000001, 2020-03-09T00:25:14.037 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.037 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.037 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809490+0000", 2020-03-09T00:25:14.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.038 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809490+0000", 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7173000000000002e-05 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809557+0000", 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1239000000000001e-05 2020-03-09T00:25:14.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.040 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809578+0000", 2020-03-09T00:25:14.040 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5971e-05 2020-03-09T00:25:14.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.040 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809594+0000", 2020-03-09T00:25:14.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6680000000000004e-06 2020-03-09T00:25:14.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809601+0000", 2020-03-09T00:25:14.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002699417 2020-03-09T00:25:14.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.042 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812300+0000", 2020-03-09T00:25:14.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6064000000000001e-05 2020-03-09T00:25:14.043 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.043 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.043 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.043 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812316+0000", 2020-03-09T00:25:14.043 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025918120000000002 2020-03-09T00:25:14.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.044 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.044 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.814908+0000", 2020-03-09T00:25:14.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9236000000000002e-05 2020-03-09T00:25:14.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.814947+0000", 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054575800000000001 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.045 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.046 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s3 53/48 ECSubWrite(tid=30, reqid=client.4751.0:346, at_version=53'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:14.046 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:14.046 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.120050055, 2020-03-09T00:25:14.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059893999999999998, 2020-03-09T00:25:14.046 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.046 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000139486 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827035+0000", 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5199999999999999e-05 2020-03-09T00:25:14.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827061+0000", 2020-03-09T00:25:14.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376999999999999e-05 2020-03-09T00:25:14.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827076+0000", 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8279999999999997e-06 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827083+0000", 2020-03-09T00:25:14.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050597700000000001 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827589+0000", 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2581e-05 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827601+0000", 2020-03-09T00:25:14.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052374839999999997 2020-03-09T00:25:14.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.052 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.052 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832839+0000", 2020-03-09T00:25:14.052 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6467000000000001e-05 2020-03-09T00:25:14.053 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.053 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.053 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.053 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832885+0000", 2020-03-09T00:25:14.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059893999999999998 2020-03-09T00:25:14.053 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 53/32 ECSubWrite(tid=65, reqid=client.4751.0:340, at_version=0'0, trim_to=0'0, roll_forward_to=53'39))", 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.827745+0000", 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.119201348000001, 2020-03-09T00:25:14.054 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048693369999999996, 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827745+0000", 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827745+0000", 2020-03-09T00:25:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8028000000000005e-05 2020-03-09T00:25:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827833+0000", 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4432e-05 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827847+0000", 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6050999999999999e-05 2020-03-09T00:25:14.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827863+0000", 2020-03-09T00:25:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4009999999999997e-06 2020-03-09T00:25:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827871+0000", 2020-03-09T00:25:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039738500000000001 2020-03-09T00:25:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831845+0000", 2020-03-09T00:25:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4450000000000006e-05 2020-03-09T00:25:14.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831930+0000", 2020-03-09T00:25:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00063670299999999997 2020-03-09T00:25:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832567+0000", 2020-03-09T00:25:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7422000000000001e-05 2020-03-09T00:25:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832614+0000", 2020-03-09T00:25:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048693369999999996 2020-03-09T00:25:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s3 53/48 ECSubWrite(tid=31, reqid=client.4751.0:347, at_version=53'19, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.834371+0000", 2020-03-09T00:25:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.112574873, 2020-03-09T00:25:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054921900000000001, 2020-03-09T00:25:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.064 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834371+0000", 2020-03-09T00:25:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.066 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834371+0000", 2020-03-09T00:25:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014995699999999999 2020-03-09T00:25:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834521+0000", 2020-03-09T00:25:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0777000000000001e-05 2020-03-09T00:25:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834542+0000", 2020-03-09T00:25:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9723000000000001e-05 2020-03-09T00:25:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834562+0000", 2020-03-09T00:25:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.359e-05 2020-03-09T00:25:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834575+0000", 2020-03-09T00:25:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8232000000000007e-05 2020-03-09T00:25:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834643+0000", 2020-03-09T00:25:14.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3478999999999999e-05 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834657+0000", 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005132803 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.839790+0000", 2020-03-09T00:25:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3628999999999996e-05 2020-03-09T00:25:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.839863+0000", 2020-03-09T00:25:14.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054921900000000001 2020-03-09T00:25:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 54/48 ECSubWrite(tid=7, reqid=client.4751.0:375, at_version=54'16, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:14.074 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.804341+0000", 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.142605015999999, 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046906919999999998, 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804341+0000", 2020-03-09T00:25:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804341+0000", 2020-03-09T00:25:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00038981900000000002 2020-03-09T00:25:14.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804731+0000", 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9690999999999998e-05 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804750+0000", 2020-03-09T00:25:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0635000000000001e-05 2020-03-09T00:25:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804761+0000", 2020-03-09T00:25:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4319999999999998e-06 2020-03-09T00:25:14.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804766+0000", 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1458999999999996e-05 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804828+0000", 2020-03-09T00:25:14.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5014999999999999e-05 2020-03-09T00:25:14.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.804843+0000", 2020-03-09T00:25:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041352130000000004 2020-03-09T00:25:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.808978+0000", 2020-03-09T00:25:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3427999999999997e-05 2020-03-09T00:25:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809032+0000", 2020-03-09T00:25:14.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046906919999999998 2020-03-09T00:25:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 54/48 ECSubWrite(tid=8, reqid=client.4751.0:376, at_version=54'17, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.13753524, 2020-03-09T00:25:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002, 2020-03-09T00:25:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.085 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000509736 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809920+0000", 2020-03-09T00:25:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8026999999999999e-05 2020-03-09T00:25:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809938+0000", 2020-03-09T00:25:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829e-05 2020-03-09T00:25:14.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809954+0000", 2020-03-09T00:25:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8799999999999997e-06 2020-03-09T00:25:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809963+0000", 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3013000000000003e-05 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810006+0000", 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2413e-05 2020-03-09T00:25:14.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810019+0000", 2020-03-09T00:25:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067421620000000003 2020-03-09T00:25:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816761+0000", 2020-03-09T00:25:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3118999999999997e-05 2020-03-09T00:25:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816814+0000", 2020-03-09T00:25:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002 2020-03-09T00:25:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.092 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 54/32 ECSubWrite(tid=77, reqid=client.4751.0:415, at_version=54'44, trim_to=0'0, roll_forward_to=54'42))", 2020-03-09T00:25:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:04.202871+0000", 2020-03-09T00:25:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.7440744699999993, 2020-03-09T00:25:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047163730000000003, 2020-03-09T00:25:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.202871+0000", 2020-03-09T00:25:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.094 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.202871+0000", 2020-03-09T00:25:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00051716700000000002 2020-03-09T00:25:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.203389+0000", 2020-03-09T00:25:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5696e-05 2020-03-09T00:25:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.203404+0000", 2020-03-09T00:25:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0539e-05 2020-03-09T00:25:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.203415+0000", 2020-03-09T00:25:14.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3109999999999999e-06 2020-03-09T00:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.203419+0000", 2020-03-09T00:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6575000000000001e-05 2020-03-09T00:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.099 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.203456+0000", 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0359999999999995e-06 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.203465+0000", 2020-03-09T00:25:14.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040849969999999999 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.207550+0000", 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8052000000000001e-05 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:04.207588+0000", 2020-03-09T00:25:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0047163730000000003 2020-03-09T00:25:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 59/48 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=59'30, trim_to=0'0, roll_forward_to=58'28))", 2020-03-09T00:25:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.134341569, 2020-03-09T00:25:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067353739999999997, 2020-03-09T00:25:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.103 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4576000000000001e-05 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812659+0000", 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0911999999999998e-05 2020-03-09T00:25:14.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812690+0000", 2020-03-09T00:25:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4375999999999999e-05 2020-03-09T00:25:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.106 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812704+0000", 2020-03-09T00:25:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2119999999999997e-06 2020-03-09T00:25:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812711+0000", 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016912599999999999 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812881+0000", 2020-03-09T00:25:14.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.377e-05 2020-03-09T00:25:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812894+0000", 2020-03-09T00:25:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064052889999999998 2020-03-09T00:25:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819300+0000", 2020-03-09T00:25:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0113e-05 2020-03-09T00:25:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819340+0000", 2020-03-09T00:25:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067353739999999997 2020-03-09T00:25:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.111 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 59/54 ECSubWrite(tid=25, reqid=osd.0.0:25, at_version=59'34, trim_to=0'0, roll_forward_to=58'32))", 2020-03-09T00:25:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.814136+0000", 2020-03-09T00:25:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.1328103919999997, 2020-03-09T00:25:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051546980000000001, 2020-03-09T00:25:14.112 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.814136+0000", 2020-03-09T00:25:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.814136+0000", 2020-03-09T00:25:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8471999999999999e-05 2020-03-09T00:25:14.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.814194+0000", 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5933999999999999e-05 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.814220+0000", 2020-03-09T00:25:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4827000000000001e-05 2020-03-09T00:25:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.814235+0000", 2020-03-09T00:25:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7660000000000001e-06 2020-03-09T00:25:14.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.814242+0000", 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094769299999999999 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815189+0000", 2020-03-09T00:25:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3008e-05 2020-03-09T00:25:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815222+0000", 2020-03-09T00:25:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040300500000000003 2020-03-09T00:25:14.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819252+0000", 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7948000000000003e-05 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819290+0000", 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0051546980000000001 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 59/54 ECSubWrite(tid=26, reqid=osd.0.0:26, at_version=59'36, trim_to=0'0, roll_forward_to=58'32))", 2020-03-09T00:25:14.120 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.815278+0000", 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.1316681160000002, 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052772160000000004, 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815278+0000", 2020-03-09T00:25:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815278+0000", 2020-03-09T00:25:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7578999999999998e-05 2020-03-09T00:25:14.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815345+0000", 2020-03-09T00:25:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6018999999999999e-05 2020-03-09T00:25:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.124 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815371+0000", 2020-03-09T00:25:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8675e-05 2020-03-09T00:25:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815390+0000", 2020-03-09T00:25:14.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1435e-05 2020-03-09T00:25:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815402+0000", 2020-03-09T00:25:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014600189999999999 2020-03-09T00:25:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.816862+0000", 2020-03-09T00:25:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2177999999999999e-05 2020-03-09T00:25:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.816874+0000", 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036416690000000002 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.820515+0000", 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9641999999999998e-05 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.820555+0000", 2020-03-09T00:25:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052772160000000004 2020-03-09T00:25:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:507 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 54] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.0422564919999999, 2020-03-09T00:25:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001, 2020-03-09T00:25:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 507 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:14.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4511999999999997e-05 2020-03-09T00:25:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904734+0000", 2020-03-09T00:25:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3980000000000004e-06 2020-03-09T00:25:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904740+0000", 2020-03-09T00:25:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4293e-05 2020-03-09T00:25:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904755+0000", 2020-03-09T00:25:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4449999999999996e-06 2020-03-09T00:25:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904763+0000", 2020-03-09T00:25:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0075999999999999e-05 2020-03-09T00:25:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904823+0000", 2020-03-09T00:25:14.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8325999999999994e-05 2020-03-09T00:25:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904911+0000", 2020-03-09T00:25:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091150910000000005 2020-03-09T00:25:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914027+0000", 2020-03-09T00:25:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097858200000000006 2020-03-09T00:25:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923812+0000", 2020-03-09T00:25:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072064440000000002 2020-03-09T00:25:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931019+0000", 2020-03-09T00:25:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6091000000000001e-05 2020-03-09T00:25:14.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931085+0000", 2020-03-09T00:25:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001 2020-03-09T00:25:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.140 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:508 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:14.141 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.042144424, 2020-03-09T00:25:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261, 2020-03-09T00:25:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:14.142 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 508 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:14.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4413000000000002e-05 2020-03-09T00:25:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904826+0000", 2020-03-09T00:25:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1389999999999997e-06 2020-03-09T00:25:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904830+0000", 2020-03-09T00:25:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3339999999999996e-06 2020-03-09T00:25:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904837+0000", 2020-03-09T00:25:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8739999999999999e-06 2020-03-09T00:25:14.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904840+0000", 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000222303 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905063+0000", 2020-03-09T00:25:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5607999999999999e-05 2020-03-09T00:25:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905128+0000", 2020-03-09T00:25:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018331841000000001 2020-03-09T00:25:14.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923460+0000", 2020-03-09T00:25:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012596800000000001 2020-03-09T00:25:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923586+0000", 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0084087399999999996 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931995+0000", 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010665 2020-03-09T00:25:14.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932101+0000", 2020-03-09T00:25:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012239099999999999 2020-03-09T00:25:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932224+0000", 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.152 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 64/32 ECSubWrite(tid=119, reqid=osd.8.0:118, at_version=0'0, trim_to=0'0, roll_forward_to=64'83))", 2020-03-09T00:25:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:13.783969+0000", 2020-03-09T00:25:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.16297669300000001, 2020-03-09T00:25:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054809639999999996, 2020-03-09T00:25:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:14.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.783969+0000", 2020-03-09T00:25:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:14.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.783969+0000", 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0179000000000001e-05 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784029+0000", 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.307e-05 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784042+0000", 2020-03-09T00:25:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5985e-05 2020-03-09T00:25:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:14.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784058+0000", 2020-03-09T00:25:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9649999999999997e-06 2020-03-09T00:25:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784067+0000", 2020-03-09T00:25:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029749799999999999 2020-03-09T00:25:14.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787042+0000", 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5451000000000002e-05 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787058+0000", 2020-03-09T00:25:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023299979999999998 2020-03-09T00:25:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.789388+0000", 2020-03-09T00:25:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2335999999999994e-05 2020-03-09T00:25:14.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.789450+0000", 2020-03-09T00:25:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054809639999999996 2020-03-09T00:25:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:14.161 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:14.161 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:14.162 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:14.161+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:14.163 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:25:14.263 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:14.261+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:14.290 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:14.290 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:14.291 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:14.291 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:14.299 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:25:14.364 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:14.362+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:14.414 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:14.414 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:14.414 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:14.414 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:14.414 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:14.422 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:25:14.464 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:14.462+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:14.538 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:14.538 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:14.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:14.539 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:14.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.539 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:14.539 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:14.539 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 20.684598173000001, 2020-03-09T00:25:14.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999, 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.541 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.541 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:14.541 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017980700000000001 2020-03-09T00:25:14.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851560+0000", 2020-03-09T00:25:14.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5363000000000001e-05 2020-03-09T00:25:14.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.543 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.543 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851585+0000", 2020-03-09T00:25:14.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4187000000000001e-05 2020-03-09T00:25:14.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851609+0000", 2020-03-09T00:25:14.544 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2890000000000003e-06 2020-03-09T00:25:14.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851618+0000", 2020-03-09T00:25:14.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00036370100000000002 2020-03-09T00:25:14.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.547 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.547 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851982+0000", 2020-03-09T00:25:14.547 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3288999999999999e-05 2020-03-09T00:25:14.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.548 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.548 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851995+0000", 2020-03-09T00:25:14.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074675740000000003 2020-03-09T00:25:14.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.549 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859463+0000", 2020-03-09T00:25:14.549 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1719999999999999e-05 2020-03-09T00:25:14.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859525+0000", 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.550 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=56, reqid=client.4751.0:254, at_version=44'23, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:25:14.551 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:14.551 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 20.672277166000001, 2020-03-09T00:25:14.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996, 2020-03-09T00:25:14.551 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.551 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.551 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.552 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:14.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.552 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.552 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:14.553 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015903600000000001 2020-03-09T00:25:14.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.553 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.553 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863860+0000", 2020-03-09T00:25:14.553 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3975999999999999e-05 2020-03-09T00:25:14.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863884+0000", 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6139999999999998e-06 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863893+0000", 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3190000000000002e-06 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863899+0000", 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039154710000000002 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867814+0000", 2020-03-09T00:25:14.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5379999999999998e-05 2020-03-09T00:25:14.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867830+0000", 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001627606 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869457+0000", 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7565999999999998e-05 2020-03-09T00:25:14.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.558 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.558 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869505+0000", 2020-03-09T00:25:14.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996 2020-03-09T00:25:14.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.558 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=77, reqid=client.4751.0:301, at_version=49'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.705279343000001, 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001, 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.559 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:14.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000394366 2020-03-09T00:25:14.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.561 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831093+0000", 2020-03-09T00:25:14.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6948999999999999e-05 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831120+0000", 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7549000000000001e-05 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831137+0000", 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1499999999999999e-06 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831146+0000", 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4503000000000001e-05 2020-03-09T00:25:14.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.564 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.564 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831230+0000", 2020-03-09T00:25:14.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4256999999999999e-05 2020-03-09T00:25:14.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.564 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831244+0000", 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014134180999999999 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845379+0000", 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5120000000000002e-05 2020-03-09T00:25:14.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.566 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.566 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845424+0000", 2020-03-09T00:25:14.566 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001 2020-03-09T00:25:14.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.566 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.567 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=78, reqid=client.4751.0:302, at_version=49'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:14.567 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:14.567 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.70068219, 2020-03-09T00:25:14.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621, 2020-03-09T00:25:14.567 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.568 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.568 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:14.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166248 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.569 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835462+0000", 2020-03-09T00:25:14.570 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6983000000000001e-05 2020-03-09T00:25:14.570 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.570 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.570 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835499+0000", 2020-03-09T00:25:14.570 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1426000000000001e-05 2020-03-09T00:25:14.570 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835520+0000", 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1362000000000001e-05 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.571 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835532+0000", 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001298925 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836831+0000", 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4944e-05 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.573 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.573 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836846+0000", 2020-03-09T00:25:14.573 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086294679999999995 2020-03-09T00:25:14.573 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.573 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.573 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.573 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845475+0000", 2020-03-09T00:25:14.574 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7265000000000002e-05 2020-03-09T00:25:14.575 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.575 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.575 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.575 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845522+0000", 2020-03-09T00:25:14.575 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621 2020-03-09T00:25:14.575 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.576 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.576 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.576 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.576 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.577 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=79, reqid=client.4751.0:303, at_version=49'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:14.577 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:14.577 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.698476502, 2020-03-09T00:25:14.577 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008, 2020-03-09T00:25:14.577 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.577 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.578 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.578 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.579 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.579 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:14.579 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.579 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.579 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.579 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.580 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:14.580 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0375999999999999e-05 2020-03-09T00:25:14.580 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.580 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837562+0000", 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1155e-05 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837583+0000", 2020-03-09T00:25:14.581 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.573e-05 2020-03-09T00:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837599+0000", 2020-03-09T00:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6429999999999998e-06 2020-03-09T00:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837605+0000", 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074652130000000001 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845071+0000", 2020-03-09T00:25:14.583 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3423e-05 2020-03-09T00:25:14.584 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.584 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.584 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.584 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845084+0000", 2020-03-09T00:25:14.584 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468407 2020-03-09T00:25:14.584 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.584 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845552+0000", 2020-03-09T00:25:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6518000000000002e-05 2020-03-09T00:25:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.585 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845619+0000", 2020-03-09T00:25:14.586 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008 2020-03-09T00:25:14.586 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.586 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.586 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.586 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.586 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.587 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/48 ECSubWrite(tid=80, reqid=client.4751.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=49'30))", 2020-03-09T00:25:14.587 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:14.587 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.696609773, 2020-03-09T00:25:14.587 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999, 2020-03-09T00:25:14.587 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.587 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.587 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7139999999999998e-05 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839425+0000", 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6959999999999999e-05 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839452+0000", 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2141000000000003e-05 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839484+0000", 2020-03-09T00:25:14.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2400000000000004e-06 2020-03-09T00:25:14.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.591 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.591 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.591 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839491+0000", 2020-03-09T00:25:14.591 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4830000000000002e-05 2020-03-09T00:25:14.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.591 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839545+0000", 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3210000000000001e-05 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839559+0000", 2020-03-09T00:25:14.592 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0054244760000000001 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.844983+0000", 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9245000000000005e-05 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.593 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.594 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845052+0000", 2020-03-09T00:25:14.594 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999 2020-03-09T00:25:14.594 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.594 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.594 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.595 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.595 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.595 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 53/52 ECSubWrite(tid=50, reqid=client.4751.0:311, at_version=53'13, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:14.595 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:14.596 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.747693658999999, 2020-03-09T00:25:14.596 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444, 2020-03-09T00:25:14.596 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.596 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.596 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.596 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.597 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.597 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:14.597 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.597 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000284932 2020-03-09T00:25:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.598 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788569+0000", 2020-03-09T00:25:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8404e-05 2020-03-09T00:25:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.599 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788608+0000", 2020-03-09T00:25:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3697e-05 2020-03-09T00:25:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.600 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788631+0000", 2020-03-09T00:25:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1178e-05 2020-03-09T00:25:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.601 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788642+0000", 2020-03-09T00:25:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000149171 2020-03-09T00:25:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.602 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.603 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788792+0000", 2020-03-09T00:25:14.605 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7829000000000002e-05 2020-03-09T00:25:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788809+0000", 2020-03-09T00:25:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0065850479999999996 2020-03-09T00:25:14.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795395+0000", 2020-03-09T00:25:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5185000000000001e-05 2020-03-09T00:25:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.607 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795450+0000", 2020-03-09T00:25:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444 2020-03-09T00:25:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.74303394, 2020-03-09T00:25:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999, 2020-03-09T00:25:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.609 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.610 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000391498 2020-03-09T00:25:14.611 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.612 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.612 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793335+0000", 2020-03-09T00:25:14.612 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2011999999999998e-05 2020-03-09T00:25:14.612 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.613 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.613 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793367+0000", 2020-03-09T00:25:14.613 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4936000000000001e-05 2020-03-09T00:25:14.613 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.613 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.613 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.613 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793382+0000", 2020-03-09T00:25:14.614 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0244000000000001e-05 2020-03-09T00:25:14.614 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.614 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.614 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.614 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793393+0000", 2020-03-09T00:25:14.614 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011101499999999999 2020-03-09T00:25:14.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.615 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.615 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.615 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793504+0000", 2020-03-09T00:25:14.615 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5529e-05 2020-03-09T00:25:14.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.616 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.616 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.616 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793519+0000", 2020-03-09T00:25:14.616 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057666640000000003 2020-03-09T00:25:14.616 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.617 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.617 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.617 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799286+0000", 2020-03-09T00:25:14.617 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011186100000000001 2020-03-09T00:25:14.617 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.617 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.617 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.618 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799398+0000", 2020-03-09T00:25:14.618 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999 2020-03-09T00:25:14.618 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.618 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.618 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.619 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.619 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:14.619 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:14.619 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.685195583, 2020-03-09T00:25:14.619 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001, 2020-03-09T00:25:14.619 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.619 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.620 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.620 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.620 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.620 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:14.620 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.620 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.620 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.621 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.621 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:14.621 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001481797 2020-03-09T00:25:14.621 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.622 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.622 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.622 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852264+0000", 2020-03-09T00:25:14.622 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3690999999999999e-05 2020-03-09T00:25:14.622 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.622 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.622 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.623 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852308+0000", 2020-03-09T00:25:14.623 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.057e-05 2020-03-09T00:25:14.623 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.623 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.623 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.623 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852338+0000", 2020-03-09T00:25:14.623 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0482999999999999e-05 2020-03-09T00:25:14.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.624 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.624 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.624 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852349+0000", 2020-03-09T00:25:14.624 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5451999999999994e-05 2020-03-09T00:25:14.624 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.625 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.625 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.625 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852434+0000", 2020-03-09T00:25:14.625 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5016e-05 2020-03-09T00:25:14.625 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.625 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.625 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.626 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852449+0000", 2020-03-09T00:25:14.626 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022365531000000001 2020-03-09T00:25:14.626 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.626 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.626 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.626 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874815+0000", 2020-03-09T00:25:14.627 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7504999999999999e-05 2020-03-09T00:25:14.627 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.627 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.627 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.627 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874872+0000", 2020-03-09T00:25:14.627 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001 2020-03-09T00:25:14.627 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.628 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.628 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.628 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.628 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.628 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=93, reqid=client.4751.0:380, at_version=54'42, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:14.628 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:14.628 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.717940354, 2020-03-09T00:25:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001, 2020-03-09T00:25:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.629 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:14.630 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.630 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.630 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.630 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.630 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:14.630 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000153535 2020-03-09T00:25:14.630 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.631 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.631 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.631 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818191+0000", 2020-03-09T00:25:14.631 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6421e-05 2020-03-09T00:25:14.632 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.633 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.633 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.633 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818218+0000", 2020-03-09T00:25:14.633 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7093000000000001e-05 2020-03-09T00:25:14.633 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.633 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.634 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.634 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818235+0000", 2020-03-09T00:25:14.634 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7960000000000001e-06 2020-03-09T00:25:14.634 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.634 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.634 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.635 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818243+0000", 2020-03-09T00:25:14.635 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011362 2020-03-09T00:25:14.635 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.635 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.635 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.635 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818357+0000", 2020-03-09T00:25:14.635 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2011000000000001e-05 2020-03-09T00:25:14.636 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.636 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.636 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.636 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818369+0000", 2020-03-09T00:25:14.636 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0092246010000000007 2020-03-09T00:25:14.637 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.637 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.637 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.637 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827594+0000", 2020-03-09T00:25:14.637 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1653000000000002e-05 2020-03-09T00:25:14.637 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.638 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.638 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.638 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827645+0000", 2020-03-09T00:25:14.638 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001 2020-03-09T00:25:14.638 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.638 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.638 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.639 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.639 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.639 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=94, reqid=client.4751.0:381, at_version=54'43, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:14.639 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:14.639 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.716645595999999, 2020-03-09T00:25:14.639 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005, 2020-03-09T00:25:14.639 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.640 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.640 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.640 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.640 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.640 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:14.640 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.640 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.641 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.641 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.641 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:14.641 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018645000000000001 2020-03-09T00:25:14.641 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.642 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.642 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.642 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819519+0000", 2020-03-09T00:25:14.642 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3156e-05 2020-03-09T00:25:14.642 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.642 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.642 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.643 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819542+0000", 2020-03-09T00:25:14.643 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5676e-05 2020-03-09T00:25:14.643 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.643 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.644 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.644 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819558+0000", 2020-03-09T00:25:14.644 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.742e-06 2020-03-09T00:25:14.644 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.644 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.645 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.645 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819564+0000", 2020-03-09T00:25:14.645 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027410770000000002 2020-03-09T00:25:14.645 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.645 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.646 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.646 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822305+0000", 2020-03-09T00:25:14.646 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5634999999999999e-05 2020-03-09T00:25:14.646 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.646 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.646 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.647 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822321+0000", 2020-03-09T00:25:14.647 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0053474569999999999 2020-03-09T00:25:14.647 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.647 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.647 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.648 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827669+0000", 2020-03-09T00:25:14.648 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4604999999999998e-05 2020-03-09T00:25:14.648 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.648 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.648 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.648 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827703+0000", 2020-03-09T00:25:14.649 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005 2020-03-09T00:25:14.649 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.649 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.649 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.649 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.649 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.649 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=98, reqid=client.4751.0:382, at_version=54'44, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:14.650 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.824765+0000", 2020-03-09T00:25:14.650 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.71121245, 2020-03-09T00:25:14.650 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048610980000000003, 2020-03-09T00:25:14.650 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.650 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.650 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.651 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.651 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.651 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.824765+0000", 2020-03-09T00:25:14.651 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.651 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.651 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.652 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.652 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.824765+0000", 2020-03-09T00:25:14.652 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000105092 2020-03-09T00:25:14.652 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.652 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.652 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.653 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.824871+0000", 2020-03-09T00:25:14.653 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.794e-05 2020-03-09T00:25:14.653 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.654 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.654 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.654 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.824899+0000", 2020-03-09T00:25:14.654 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8057999999999999e-05 2020-03-09T00:25:14.654 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.654 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.654 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.655 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.824917+0000", 2020-03-09T00:25:14.656 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8280000000000003e-06 2020-03-09T00:25:14.656 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.656 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.656 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.656 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.824924+0000", 2020-03-09T00:25:14.657 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001818568 2020-03-09T00:25:14.657 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.657 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.657 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.657 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.826743+0000", 2020-03-09T00:25:14.657 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.838e-05 2020-03-09T00:25:14.657 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.658 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.658 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.658 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.826761+0000", 2020-03-09T00:25:14.658 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0028008199999999999 2020-03-09T00:25:14.658 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.658 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.658 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.659 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829562+0000", 2020-03-09T00:25:14.659 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.4412000000000006e-05 2020-03-09T00:25:14.659 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.659 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.659 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.659 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829627+0000", 2020-03-09T00:25:14.660 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048610980000000003 2020-03-09T00:25:14.660 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.660 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.660 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.660 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.660 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.660 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 54/32 ECSubWrite(tid=105, reqid=client.4751.0:427, at_version=54'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:14.661 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:14.661 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.313227041999999, 2020-03-09T00:25:14.661 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008, 2020-03-09T00:25:14.661 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.661 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.661 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.661 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:14.662 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017923099999999999 2020-03-09T00:25:14.663 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.663 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.663 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.663 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222930+0000", 2020-03-09T00:25:14.663 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4340000000000001e-05 2020-03-09T00:25:14.663 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.664 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.664 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.665 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222954+0000", 2020-03-09T00:25:14.665 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4165000000000001e-05 2020-03-09T00:25:14.665 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.665 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.665 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.665 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222969+0000", 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6719999999999993e-06 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222977+0000", 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7257999999999998e-05 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.666 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.667 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.667 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223055+0000", 2020-03-09T00:25:14.667 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4771e-05 2020-03-09T00:25:14.667 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.667 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.667 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.667 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223069+0000", 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.008721477 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231791+0000", 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3718000000000003e-05 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.668 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.669 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.669 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231845+0000", 2020-03-09T00:25:14.669 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008 2020-03-09T00:25:14.669 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.669 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.669 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.669 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.670 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.670 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 54/32 ECSubWrite(tid=106, reqid=client.4751.0:428, at_version=54'19, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:14.670 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.227034+0000", 2020-03-09T00:25:14.670 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 10.308943447000001, 2020-03-09T00:25:14.670 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0051962049999999997, 2020-03-09T00:25:14.670 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.670 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.671 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.671 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.671 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.671 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227034+0000", 2020-03-09T00:25:14.671 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.671 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.671 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.672 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.672 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227034+0000", 2020-03-09T00:25:14.672 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016866400000000001 2020-03-09T00:25:14.672 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.672 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.672 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.672 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227203+0000", 2020-03-09T00:25:14.673 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0024999999999999e-05 2020-03-09T00:25:14.673 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.673 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.673 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.673 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227223+0000", 2020-03-09T00:25:14.673 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7070000000000001e-05 2020-03-09T00:25:14.673 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.674 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.674 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.674 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227240+0000", 2020-03-09T00:25:14.674 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5740000000000001e-06 2020-03-09T00:25:14.675 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.675 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.675 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.675 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227247+0000", 2020-03-09T00:25:14.675 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2106000000000004e-05 2020-03-09T00:25:14.676 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.676 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.676 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.676 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227319+0000", 2020-03-09T00:25:14.676 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5736e-05 2020-03-09T00:25:14.677 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.677 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.677 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.677 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.227335+0000", 2020-03-09T00:25:14.677 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004863844 2020-03-09T00:25:14.677 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.677 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.232198+0000", 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2186e-05 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.232231+0000", 2020-03-09T00:25:14.678 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0051962049999999997 2020-03-09T00:25:14.679 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.679 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.679 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.679 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.679 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.679 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 58/54 ECSubWrite(tid=22, reqid=osd.0.0:22, at_version=58'30, trim_to=0'0, roll_forward_to=57'28))", 2020-03-09T00:25:14.679 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:07.813895+0000", 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.722082726, 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0052904120000000004, 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.680 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813895+0000", 2020-03-09T00:25:14.681 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.681 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.681 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.681 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.681 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813895+0000", 2020-03-09T00:25:14.681 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2805000000000006e-05 2020-03-09T00:25:14.681 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813958+0000", 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1939e-05 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.682 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.813990+0000", 2020-03-09T00:25:14.683 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5857000000000001e-05 2020-03-09T00:25:14.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.683 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.683 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.683 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.814006+0000", 2020-03-09T00:25:14.683 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6449999999999999e-06 2020-03-09T00:25:14.683 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.814012+0000", 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010552600000000001 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.684 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.814118+0000", 2020-03-09T00:25:14.685 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2829e-05 2020-03-09T00:25:14.685 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.685 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.685 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.685 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.814131+0000", 2020-03-09T00:25:14.685 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0049987709999999999 2020-03-09T00:25:14.685 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.819130+0000", 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6039999999999999e-05 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.686 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.819186+0000", 2020-03-09T00:25:14.687 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0052904120000000004 2020-03-09T00:25:14.687 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.687 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.687 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.687 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.687 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.688 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/32 ECSubWrite(tid=162, reqid=osd.11.0:162, at_version=63'46, trim_to=0'0, roll_forward_to=60'44))", 2020-03-09T00:25:14.688 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:14.688 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.7593525350000001, 2020-03-09T00:25:14.688 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001, 2020-03-09T00:25:14.688 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.688 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.688 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.689 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:14.690 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5686999999999997e-05 2020-03-09T00:25:14.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.690 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.690 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.690 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776681+0000", 2020-03-09T00:25:14.690 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2730000000000001e-05 2020-03-09T00:25:14.690 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776704+0000", 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4839e-05 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.691 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776719+0000", 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7049999999999999e-06 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776726+0000", 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00107576 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.692 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.693 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.693 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777802+0000", 2020-03-09T00:25:14.693 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2907e-05 2020-03-09T00:25:14.693 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.693 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.693 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.693 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777815+0000", 2020-03-09T00:25:14.694 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0051422669999999998 2020-03-09T00:25:14.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.694 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.694 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.694 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782957+0000", 2020-03-09T00:25:14.694 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1214e-05 2020-03-09T00:25:14.694 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.695 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.695 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.695 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.783019+0000", 2020-03-09T00:25:14.695 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001 2020-03-09T00:25:14.695 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.695 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 63/32 ECSubWrite(tid=110, reqid=client.4751.0:522, at_version=63'73, trim_to=0'0, roll_forward_to=60'71))", 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.7588645510000001, 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001, 2020-03-09T00:25:14.696 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.697 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.697 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.697 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.697 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.697 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:14.697 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7158999999999999e-05 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.698 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.699 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777171+0000", 2020-03-09T00:25:14.699 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0669000000000001e-05 2020-03-09T00:25:14.699 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.699 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.699 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.699 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777191+0000", 2020-03-09T00:25:14.699 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5477000000000001e-05 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777207+0000", 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2890000000000003e-06 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.700 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.701 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777213+0000", 2020-03-09T00:25:14.701 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010217600000000001 2020-03-09T00:25:14.701 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.701 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.701 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.701 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777315+0000", 2020-03-09T00:25:14.701 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2303999999999999e-05 2020-03-09T00:25:14.702 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.702 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.702 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.702 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777327+0000", 2020-03-09T00:25:14.702 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.005510989 2020-03-09T00:25:14.702 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.703 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.703 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782838+0000", 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7664000000000003e-05 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782886+0000", 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001 2020-03-09T00:25:14.704 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.705 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.705 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.705 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.705 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.705 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 63/54 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=63'40, trim_to=0'0, roll_forward_to=60'38))", 2020-03-09T00:25:14.705 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:14.705 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.757417314, 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004, 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:14.706 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6827999999999999e-05 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.707 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.708 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778617+0000", 2020-03-09T00:25:14.708 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1481999999999999e-05 2020-03-09T00:25:14.708 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.708 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.708 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.708 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778639+0000", 2020-03-09T00:25:14.708 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4163e-05 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778653+0000", 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3899999999999998e-06 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.709 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778659+0000", 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7515000000000004e-05 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778757+0000", 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2418e-05 2020-03-09T00:25:14.710 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.711 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778769+0000", 2020-03-09T00:25:14.711 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058080179999999999 2020-03-09T00:25:14.711 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.711 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.711 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784577+0000", 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3770000000000002e-05 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784631+0000", 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004 2020-03-09T00:25:14.712 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.713 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.713 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.713 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.713 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.713 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 63/54 ECSubWrite(tid=85, reqid=osd.14.0:85, at_version=63'28, trim_to=0'0, roll_forward_to=62'26))", 2020-03-09T00:25:14.713 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:14.713 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.756093511, 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001, 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:14.714 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9020000000000001e-05 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.715 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779943+0000", 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8972e-05 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779972+0000", 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5282999999999999e-05 2020-03-09T00:25:14.716 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.717 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.717 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.717 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779988+0000", 2020-03-09T00:25:14.717 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5200000000000003e-06 2020-03-09T00:25:14.717 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.717 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.717 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779994+0000", 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027147400000000001 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780266+0000", 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1885000000000001e-05 2020-03-09T00:25:14.718 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780278+0000", 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0050865659999999998 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785364+0000", 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6227999999999999e-05 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785410+0000", 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.720 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.721 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.721 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.721 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.721 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 63/54 ECSubWrite(tid=86, reqid=osd.14.0:86, at_version=63'30, trim_to=0'0, roll_forward_to=62'26))", 2020-03-09T00:25:14.721 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.780051+0000", 2020-03-09T00:25:14.721 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 1.7559267999999999, 2020-03-09T00:25:14.721 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0054387860000000001, 2020-03-09T00:25:14.722 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:14.722 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:14.722 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:14.722 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.722 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:14.722 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780051+0000", 2020-03-09T00:25:14.722 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780051+0000", 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5777000000000004e-05 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.723 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780117+0000", 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1569999999999998e-05 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780138+0000", 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4792e-05 2020-03-09T00:25:14.724 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.725 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.725 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:14.725 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780153+0000", 2020-03-09T00:25:14.725 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6969999999999998e-06 2020-03-09T00:25:14.725 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.725 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.725 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780159+0000", 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004356309 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784515+0000", 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.115e-05 2020-03-09T00:25:14.726 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784526+0000", 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00093218200000000004 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:14.727 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785459+0000", 2020-03-09T00:25:14.728 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1309000000000001e-05 2020-03-09T00:25:14.728 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:14.728 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:14.728 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:14.728 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785490+0000", 2020-03-09T00:25:14.728 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0054387860000000001 2020-03-09T00:25:14.728 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.729 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.729 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.729 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:14.729 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:14.729 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:14.730 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:14.731 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:14.728+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:14.732 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:25:14.776 INFO:tasks.rados.rados.0.smithi104.stdout:213: done (6 left) 2020-03-09T00:25:14.776 INFO:tasks.rados.rados.0.smithi104.stdout:214: copy_from oid 35 from oid 12 current snap is 16 2020-03-09T00:25:14.777 INFO:tasks.rados.rados.0.smithi104.stdout:206: finishing rollback tid 2 to smithi10428818-31 2020-03-09T00:25:14.777 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 31 v 38 (ObjNum 64 snap 8 seq_num 64) dirty exists 2020-03-09T00:25:14.777 INFO:tasks.rados.rados.0.smithi104.stdout:210: finishing rollback tid 0 to smithi10428818-32 2020-03-09T00:25:14.777 INFO:tasks.rados.rados.0.smithi104.stdout:210: finishing rollback tid 1 to smithi10428818-32 2020-03-09T00:25:14.778 INFO:tasks.rados.rados.0.smithi104.stdout:210: finishing rollback tid 2 to smithi10428818-32 2020-03-09T00:25:14.778 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 32 v 54 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-03-09T00:25:14.778 INFO:tasks.rados.rados.0.smithi104.stdout:208: finishing write tid 1 to smithi10428818-7 2020-03-09T00:25:14.778 INFO:tasks.rados.rados.0.smithi104.stdout:208: finishing write tid 2 to smithi10428818-7 2020-03-09T00:25:14.778 INFO:tasks.rados.rados.0.smithi104.stdout:208: finishing write tid 3 to smithi10428818-7 2020-03-09T00:25:14.778 INFO:tasks.rados.rados.0.smithi104.stdout:209: finishing copy_from to smithi10428818-5 2020-03-09T00:25:14.778 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 5 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:208: finishing write tid 4 to smithi10428818-7 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:208: finishing write tid 6 to smithi10428818-7 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:209: finishing copy_from racing read to smithi10428818-5 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:211: finishing rollback tid 0 to smithi10428818-37 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:211: finishing rollback tid 1 to smithi10428818-37 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:211: finishing rollback tid 2 to smithi10428818-37 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 37 v 116 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-03-09T00:25:14.779 INFO:tasks.rados.rados.0.smithi104.stdout:212: finishing write tid 1 to smithi10428818-15 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:208: finishing write tid 7 to smithi10428818-7 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 7 v 113 (ObjNum 97 snap 16 seq_num 97) dirty exists 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:208: left oid 7 (ObjNum 97 snap 16 seq_num 97) 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:212: finishing write tid 2 to smithi10428818-15 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:206: done (6 left) 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:208: done (5 left) 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:209: done (4 left) 2020-03-09T00:25:14.780 INFO:tasks.rados.rados.0.smithi104.stdout:210: done (3 left) 2020-03-09T00:25:14.781 INFO:tasks.rados.rados.0.smithi104.stdout:211: done (2 left) 2020-03-09T00:25:14.781 INFO:tasks.rados.rados.0.smithi104.stdout:215: snap_create 2020-03-09T00:25:14.781 INFO:tasks.rados.rados.0.smithi104.stdout:212: finishing write tid 3 to smithi10428818-15 2020-03-09T00:25:14.781 INFO:tasks.rados.rados.0.smithi104.stdout:212: finishing write tid 4 to smithi10428818-15 2020-03-09T00:25:14.781 INFO:tasks.rados.rados.0.smithi104.stdout:212: finishing write tid 6 to smithi10428818-15 2020-03-09T00:25:14.781 INFO:tasks.rados.rados.0.smithi104.stdout:212: finishing write tid 7 to smithi10428818-15 2020-03-09T00:25:14.781 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 15 v 82 (ObjNum 98 snap 16 seq_num 98) dirty exists 2020-03-09T00:25:14.782 INFO:tasks.rados.rados.0.smithi104.stdout:212: left oid 15 (ObjNum 98 snap 16 seq_num 98) 2020-03-09T00:25:14.804 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-03-09T00:25:14.824 INFO:tasks.rados.rados.0.smithi104.stdout:214: finishing copy_from to smithi10428818-35 2020-03-09T00:25:14.824 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 35 v 24 (ObjNum 75 snap 11 seq_num 75) dirty exists 2020-03-09T00:25:14.825 INFO:tasks.rados.rados.0.smithi104.stdout:214: finishing copy_from racing read to smithi10428818-35 2020-03-09T00:25:14.831 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:14.830+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:14.879 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:14.879 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:14.879 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:14.879 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:14.889 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:25:14.893 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:14.914 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:14.933 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:14.931+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:14.935 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:14.967 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:14.987 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:15.008 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:15.013 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.013 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.022 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:25:15.028 INFO:teuthology.orchestra.run.smithi148.stdout:osd.6: {} 2020-03-09T00:25:15.033 INFO:tasks.ceph.osd.6.smithi139.stderr:2020-03-09T00:25:15.031+0000 7f464ee2b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26970) UID: 0 2020-03-09T00:25:15.051 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:15.072 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:15.094 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:15.117 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:15.134 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:15.132+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:15.135 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:15.145 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.145 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.154 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:25:15.155 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:15.155 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:15.152+0000 7fa753fff700 0 -- 172.21.15.148:0/216495087 >> [v2:172.21.15.50:6816/27631,v1:172.21.15.50:6817/27631] conn(0x7fa73802c0e0 msgr2=0x7fa73802e4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:15.242 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:15.239+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:15.271 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.271 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.280 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:25:15.343 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:15.341+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:15.396 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:15.404 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.405 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.409 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:15.413 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:25:15.431 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:15.432 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.433 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.434 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.435 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.435 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.435 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:15.444 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:15.442+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:15.530 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.530 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.539 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:25:15.544 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:15.542+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:15.646 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:15.644+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:15.661 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.662 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.670 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:25:15.747 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:15.745+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:15.780 INFO:tasks.rados.rados.0.smithi104.stdout:212: done (2 left) 2020-03-09T00:25:15.780 INFO:tasks.rados.rados.0.smithi104.stdout:214: done (1 left) 2020-03-09T00:25:15.780 INFO:tasks.rados.rados.0.smithi104.stdout:215: done (0 left) 2020-03-09T00:25:15.781 INFO:tasks.rados.rados.0.smithi104.stdout:216: snap_remove snap 7 2020-03-09T00:25:15.783 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is set 2020-03-09T00:25:15.826 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.826 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.835 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:25:15.878 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:15.877+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:15.949 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:15.948+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:15.950 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:15.950 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:15.959 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-03-09T00:25:16.050 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:16.048+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:16.081 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:16.081 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:16.091 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-03-09T00:25:16.151 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:16.150+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:16.207 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:16.207 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:16.216 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-03-09T00:25:16.252 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:16.250+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:16.332 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:16.332 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:16.341 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:16.352 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:16.351+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:16.453 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:16.452+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:16.478 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:16.478 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:16.487 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:16.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 14, 15] out_osds: [13, 5] dead_osds: [] live_osds: [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:25:16.509 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 2020-03-09T00:25:16.509 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [11, 10, 13, 12, 15, 14, 1, 0, 3, 2, 5, 4, 7, 6, 9, 8] 2020-03-09T00:25:16.509 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2020-03-09T00:25:16.510 INFO:teuthology.orchestra.run:waiting for 300 2020-03-09T00:25:16.534 INFO:tasks.ceph.osd.6:Stopped 2020-03-09T00:25:16.534 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2020-03-09T00:25:16.534 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi139.front.sepia.ceph.com 2020-03-09T00:25:16.534 INFO:teuthology.orchestra.run.smithi139:> type ceph-objectstore-tool 2020-03-09T00:25:16.574 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:16.572+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:16.587 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-03-09T00:25:16.587 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2020-03-09T00:25:16.611 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:16.611 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:16.620 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:16.663 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:16.661+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:16.744 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:16.745 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:16.753 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:25:16.756 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:16.755+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:16.766 INFO:tasks.rados.rados.0.smithi104.stdout:216: done (0 left) 2020-03-09T00:25:16.766 INFO:tasks.rados.rados.0.smithi104.stdout:217: delete oid 33 current snap is 17 2020-03-09T00:25:16.857 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:16.855+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:16.885 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:16.885 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:16.893 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:25:16.957 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:16.955+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:17.050 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:17.050 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:17.058 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:17.057+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:17.059 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:25:17.159 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:17.158+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:17.214 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:17.214 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:17.222 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:25:17.267 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:17.265+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:17.342 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:17.342 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:17.350 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:25:17.377 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:17.369+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:17.443 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:17.468 INFO:teuthology.orchestra.run.smithi139.stdout:3.ds6 2020-03-09T00:25:17.468 INFO:teuthology.orchestra.run.smithi139.stdout:3.cs6 2020-03-09T00:25:17.468 INFO:teuthology.orchestra.run.smithi139.stdout:3.bs1 2020-03-09T00:25:17.468 INFO:teuthology.orchestra.run.smithi139.stdout:3.8s3 2020-03-09T00:25:17.468 INFO:teuthology.orchestra.run.smithi139.stdout:3.7s5 2020-03-09T00:25:17.468 INFO:teuthology.orchestra.run.smithi139.stdout:3.5s1 2020-03-09T00:25:17.469 INFO:teuthology.orchestra.run.smithi139.stdout:3.4s4 2020-03-09T00:25:17.469 INFO:teuthology.orchestra.run.smithi139.stdout:3.3s5 2020-03-09T00:25:17.469 INFO:teuthology.orchestra.run.smithi139.stdout:3.1s0 2020-03-09T00:25:17.469 INFO:teuthology.orchestra.run.smithi139.stdout:3.0s6 2020-03-09T00:25:17.469 INFO:teuthology.orchestra.run.smithi139.stdout:2.7 2020-03-09T00:25:17.469 INFO:teuthology.orchestra.run.smithi139.stdout:3.9s6 2020-03-09T00:25:17.469 INFO:teuthology.orchestra.run.smithi139.stdout:3.6s4 2020-03-09T00:25:17.470 INFO:teuthology.orchestra.run.smithi139.stdout:2.4 2020-03-09T00:25:17.472 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:17.469+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:17.509 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:17.509 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:17.518 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:25:17.573 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:17.571+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:17.588 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:17.606 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:17.627 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:17.647 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:17.666 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:17.673 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:17.672+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:17.675 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:17.675 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:17.684 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:25:17.686 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:17.707 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:17.727 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:17.747 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:17.774 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:17.774 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:17.772+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:17.786 INFO:tasks.rados.rados.0.smithi104.stdout:217: done (0 left) 2020-03-09T00:25:17.786 INFO:tasks.rados.rados.0.smithi104.stdout:218: read oid 48 snap -1 2020-03-09T00:25:17.786 INFO:tasks.rados.rados.0.smithi104.stdout:218: expect (ObjNum 71 snap 11 seq_num 71) 2020-03-09T00:25:17.786 INFO:tasks.rados.rados.0.smithi104.stdout:219: read oid 2 snap -1 2020-03-09T00:25:17.786 INFO:tasks.rados.rados.0.smithi104.stdout:219: expect (ObjNum 1 snap 0 seq_num 1) 2020-03-09T00:25:17.786 INFO:tasks.rados.rados.0.smithi104.stdout:220: delete oid 7 current snap is 17 2020-03-09T00:25:17.794 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-03-09T00:25:17.795 INFO:tasks.rados.rados.0.smithi104.stdout:219: expect (ObjNum 1 snap 0 seq_num 1) 2020-03-09T00:25:17.800 INFO:tasks.rados.rados.0.smithi104.stdout:218: expect (ObjNum 71 snap 11 seq_num 71) 2020-03-09T00:25:17.806 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:17.826 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:17.826 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:17.831 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:17.835 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:25:17.859 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:17.874 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:17.873+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:17.879 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:17.901 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:17.902 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.902 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.905 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:17.912 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:17.913 INFO:tasks.rados.rados.0.smithi104.stdout:218: done (2 left) 2020-03-09T00:25:17.914 INFO:tasks.rados.rados.0.smithi104.stdout:219: done (1 left) 2020-03-09T00:25:17.914 INFO:tasks.rados.rados.0.smithi104.stdout:220: done (0 left) 2020-03-09T00:25:17.914 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 23 current snap is 17 2020-03-09T00:25:17.914 INFO:tasks.rados.rados.0.smithi104.stdout:221: seq_num 101 ranges {3964928=540672,4505600=507904,5013504=622592,5636096=32768} 2020-03-09T00:25:17.937 INFO:tasks.rados.rados.0.smithi104.stdout:221: writing smithi10428818-23 from 3964928 to 4505600 tid 1 2020-03-09T00:25:17.939 INFO:tasks.rados.rados.0.smithi104.stdout:221: writing smithi10428818-23 from 4505600 to 5013504 tid 2 2020-03-09T00:25:17.944 INFO:tasks.rados.rados.0.smithi104.stdout:221: writing smithi10428818-23 from 5013504 to 5636096 tid 3 2020-03-09T00:25:17.944 INFO:tasks.rados.rados.0.smithi104.stdout:221: writing smithi10428818-23 from 5636096 to 5668864 tid 4 2020-03-09T00:25:17.945 INFO:tasks.rados.rados.0.smithi104.stdout:222: read oid 6 snap -1 2020-03-09T00:25:17.945 INFO:tasks.rados.rados.0.smithi104.stdout:222: expect deleted 2020-03-09T00:25:17.945 INFO:tasks.rados.rados.0.smithi104.stdout:223: copy_from oid 35 from oid 24 current snap is 17 2020-03-09T00:25:17.945 INFO:tasks.rados.rados.0.smithi104.stdout:224: setattr oid 4 current snap is 17 2020-03-09T00:25:17.947 INFO:tasks.rados.rados.0.smithi104.stdout:222: done (3 left) 2020-03-09T00:25:17.947 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 11 current snap is 17 2020-03-09T00:25:17.947 INFO:tasks.rados.rados.0.smithi104.stdout:225: seq_num 102 ranges {229376=557056,786432=638976,1425408=524288} 2020-03-09T00:25:17.952 INFO:tasks.rados.rados.0.smithi104.stdout:225: writing smithi10428818-11 from 229376 to 786432 tid 1 2020-03-09T00:25:17.956 INFO:tasks.rados.rados.0.smithi104.stdout:225: writing smithi10428818-11 from 786432 to 1425408 tid 2 2020-03-09T00:25:17.959 INFO:tasks.rados.rados.0.smithi104.stdout:225: writing smithi10428818-11 from 1425408 to 1949696 tid 3 2020-03-09T00:25:17.960 INFO:tasks.rados.rados.0.smithi104.stdout:226: copy_from oid 46 from oid 49 current snap is 17 2020-03-09T00:25:17.960 INFO:tasks.rados.rados.0.smithi104.stdout:221: finishing write tid 1 to smithi10428818-23 2020-03-09T00:25:17.960 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 22 current snap is 17 2020-03-09T00:25:17.960 INFO:tasks.rados.rados.0.smithi104.stdout:227: seq_num 103 ranges {573440=573440,1146880=786432,1933312=376832} 2020-03-09T00:25:17.966 INFO:tasks.rados.rados.0.smithi104.stdout:227: writing smithi10428818-22 from 573440 to 1146880 tid 1 2020-03-09T00:25:17.970 INFO:tasks.rados.rados.0.smithi104.stdout:227: writing smithi10428818-22 from 1146880 to 1933312 tid 2 2020-03-09T00:25:17.973 INFO:tasks.rados.rados.0.smithi104.stdout:227: writing smithi10428818-22 from 1933312 to 2310144 tid 3 2020-03-09T00:25:17.973 INFO:tasks.rados.rados.0.smithi104.stdout:228: rmattr oid 41 current snap is 17 2020-03-09T00:25:17.973 INFO:tasks.rados.rados.0.smithi104.stdout:228: done (6 left) 2020-03-09T00:25:17.973 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 40 current snap is 17 2020-03-09T00:25:17.974 INFO:tasks.rados.rados.0.smithi104.stdout:229: seq_num 104 ranges {0=589824,589824=540672,1130496=622592} 2020-03-09T00:25:17.976 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:17.974+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:17.976 INFO:tasks.rados.rados.0.smithi104.stdout:229: writing smithi10428818-40 from 0 to 589824 tid 1 2020-03-09T00:25:17.981 INFO:tasks.rados.rados.0.smithi104.stdout:229: writing smithi10428818-40 from 589824 to 1130496 tid 2 2020-03-09T00:25:17.984 INFO:tasks.rados.rados.0.smithi104.stdout:229: writing smithi10428818-40 from 1130496 to 1753088 tid 3 2020-03-09T00:25:17.984 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 21 current snap is 17 2020-03-09T00:25:17.984 INFO:tasks.rados.rados.0.smithi104.stdout:230: seq_num 105 ranges {0=606208,606208=688128,1294336=475136} 2020-03-09T00:25:17.987 INFO:tasks.rados.rados.0.smithi104.stdout:230: writing smithi10428818-21 from 0 to 606208 tid 1 2020-03-09T00:25:17.988 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 2.7 --file /var/log/ceph/exp.2.7.6 2020-03-09T00:25:17.991 INFO:tasks.rados.rados.0.smithi104.stdout:230: writing smithi10428818-21 from 606208 to 1294336 tid 2 2020-03-09T00:25:17.995 INFO:tasks.rados.rados.0.smithi104.stdout:230: writing smithi10428818-21 from 1294336 to 1769472 tid 3 2020-03-09T00:25:17.995 INFO:tasks.rados.rados.0.smithi104.stdout:231: snap_remove snap 11 2020-03-09T00:25:18.011 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:18.012 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:18.019 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:25:18.077 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:18.075+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:18.148 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:18.148 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:18.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:18.148 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:18.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.149 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:2 1.0 1:e8e3cb91:::INTEL_SSDPED1D480GA_PHMB7513001F480DGN:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-03-09T00:25:18.149 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.663306+0000", 2020-03-09T00:25:18.149 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.482129456999999, 2020-03-09T00:25:18.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018269950000000001, 2020-03-09T00:25:18.149 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.149 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663306+0000", 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663306+0000", 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8458000000000002e-05 2020-03-09T00:25:18.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663365+0000", 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9190000000000004e-06 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663371+0000", 2020-03-09T00:25:18.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2291e-05 2020-03-09T00:25:18.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663383+0000", 2020-03-09T00:25:18.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6699999999999994e-06 2020-03-09T00:25:18.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663391+0000", 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0685e-05 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663441+0000", 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000138608 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663580+0000", 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013393300000000001 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0", 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.663714+0000", 2020-03-09T00:25:18.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052014900000000002 2020-03-09T00:25:18.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2020-03-09T00:25:18.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.664234+0000", 2020-03-09T00:25:18.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080904499999999999 2020-03-09T00:25:18.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2020-03-09T00:25:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665043+0000", 2020-03-09T00:25:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1595999999999999e-05 2020-03-09T00:25:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665095+0000", 2020-03-09T00:25:18.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8640999999999998e-05 2020-03-09T00:25:18.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.665133+0000", 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018269950000000001 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:4 1.0 1:a3830038:::INTEL_SSDPEDMD400G4_PHFT620500N6400BGN:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-03-09T00:25:18.162 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.745925+0000", 2020-03-09T00:25:18.163 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.399511185999998, 2020-03-09T00:25:18.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018936129999999999, 2020-03-09T00:25:18.163 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.163 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.163 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:25:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:25:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 4 2020-03-09T00:25:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745925+0000", 2020-03-09T00:25:18.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745925+0000", 2020-03-09T00:25:18.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1280999999999998e-05 2020-03-09T00:25:18.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745986+0000", 2020-03-09T00:25:18.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3940000000000004e-06 2020-03-09T00:25:18.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.745991+0000", 2020-03-09T00:25:18.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1146e-05 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746002+0000", 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9070000000000007e-06 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746010+0000", 2020-03-09T00:25:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2175000000000001e-05 2020-03-09T00:25:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746063+0000", 2020-03-09T00:25:18.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013140800000000001 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746194+0000", 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010697 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0", 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746301+0000", 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000434876 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.746736+0000", 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099432899999999996 2020-03-09T00:25:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747730+0000", 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9877999999999999e-05 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:18.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747780+0000", 2020-03-09T00:25:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8248999999999999e-05 2020-03-09T00:25:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.747818+0000", 2020-03-09T00:25:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018936129999999999 2020-03-09T00:25:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4118.0:6 1.0 1:15e1a584:::INTEL_SSDPEDMD400G4_CVFT6233005G400BGN:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-03-09T00:25:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:24.843698+0000", 2020-03-09T00:25:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.301738032000003, 2020-03-09T00:25:18.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020264290000000002, 2020-03-09T00:25:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4118", 2020-03-09T00:25:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/579184066", 2020-03-09T00:25:18.177 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2020-03-09T00:25:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843698+0000", 2020-03-09T00:25:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843698+0000", 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4163000000000003e-05 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843752+0000", 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9470000000000002e-06 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843757+0000", 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1093999999999999e-05 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843768+0000", 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4360000000000004e-06 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843772+0000", 2020-03-09T00:25:18.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1758999999999997e-05 2020-03-09T00:25:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843824+0000", 2020-03-09T00:25:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124189 2020-03-09T00:25:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.843948+0000", 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109038 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0", 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.844057+0000", 2020-03-09T00:25:18.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073162100000000001 2020-03-09T00:25:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2020-03-09T00:25:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.844789+0000", 2020-03-09T00:25:18.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000853398 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845642+0000", 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0244000000000002e-05 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845693+0000", 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.154e-05 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:24.845724+0000", 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020264290000000002 2020-03-09T00:25:18.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:171 3.3s0 3:c607ec57:::smithi10428818-30:head [read 0~491520,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:25:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.810479+0000", 2020-03-09T00:25:18.187 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.334956704, 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054941469999999996, 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 171 2020-03-09T00:25:18.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810479+0000", 2020-03-09T00:25:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810479+0000", 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1289999999999999e-05 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810520+0000", 2020-03-09T00:25:18.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1490000000000002e-06 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810526+0000", 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0669000000000001e-05 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810536+0000", 2020-03-09T00:25:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2419999999999997e-06 2020-03-09T00:25:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810542+0000", 2020-03-09T00:25:18.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6380000000000001e-05 2020-03-09T00:25:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810579+0000", 2020-03-09T00:25:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8528000000000004e-05 2020-03-09T00:25:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810657+0000", 2020-03-09T00:25:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0053158889999999999 2020-03-09T00:25:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.815973+0000", 2020-03-09T00:25:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054941469999999996 2020-03-09T00:25:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:172 3.3s0 3:c607ec57:::smithi10428818-30:head [read 0~491520,checksum] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:25:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.810578+0000", 2020-03-09T00:25:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.334857598999999, 2020-03-09T00:25:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066558590000000001, 2020-03-09T00:25:18.196 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 172 2020-03-09T00:25:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.197 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810578+0000", 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810578+0000", 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9969999999999998e-05 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810628+0000", 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7519999999999998e-06 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810632+0000", 2020-03-09T00:25:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5470000000000002e-06 2020-03-09T00:25:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810638+0000", 2020-03-09T00:25:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7400000000000002e-06 2020-03-09T00:25:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810642+0000", 2020-03-09T00:25:18.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044449300000000001 2020-03-09T00:25:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.811087+0000", 2020-03-09T00:25:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8905000000000004e-05 2020-03-09T00:25:18.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.811166+0000", 2020-03-09T00:25:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060684520000000002 2020-03-09T00:25:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.817234+0000", 2020-03-09T00:25:18.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066558590000000001 2020-03-09T00:25:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:173 3.3s0 3:c607ec57:::smithi10428818-30:head [read 0~491520,checksum] snapc 0=[] ondisk+read+known_if_redirected e37)", 2020-03-09T00:25:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.810673+0000", 2020-03-09T00:25:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.334763057, 2020-03-09T00:25:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079003400000000005, 2020-03-09T00:25:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:18.206 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 173 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810673+0000", 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810673+0000", 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9033999999999999e-05 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810712+0000", 2020-03-09T00:25:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.315e-06 2020-03-09T00:25:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810715+0000", 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2269999999999998e-06 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810721+0000", 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.418e-06 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.810724+0000", 2020-03-09T00:25:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013699579999999999 2020-03-09T00:25:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.812094+0000", 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5985e-05 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.812150+0000", 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064234030000000003 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.818573+0000", 2020-03-09T00:25:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079003400000000005 2020-03-09T00:25:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.213 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpReadReply(3.3s0 37/32 ECSubReadReply(tid=35, attrs_read=0))", 2020-03-09T00:25:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:46.816810+0000", 2020-03-09T00:25:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.328626215, 2020-03-09T00:25:18.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017827629999999999, 2020-03-09T00:25:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816810+0000", 2020-03-09T00:25:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816810+0000", 2020-03-09T00:25:18.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011718999999999999 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816927+0000", 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.093e-05 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816938+0000", 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4128999999999999e-05 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816962+0000", 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5979999999999996e-06 2020-03-09T00:25:18.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.816967+0000", 2020-03-09T00:25:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001158877 2020-03-09T00:25:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.818126+0000", 2020-03-09T00:25:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046603900000000001 2020-03-09T00:25:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:46.818592+0000", 2020-03-09T00:25:18.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017827629999999999 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s6 38/32 ECSubWrite(tid=20, reqid=client.4751.0:190, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.823178+0000", 2020-03-09T00:25:18.222 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.322257689000001, 2020-03-09T00:25:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023877159999999998, 2020-03-09T00:25:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823178+0000", 2020-03-09T00:25:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823178+0000", 2020-03-09T00:25:18.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6784999999999999e-05 2020-03-09T00:25:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823235+0000", 2020-03-09T00:25:18.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1030999999999999e-05 2020-03-09T00:25:18.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823256+0000", 2020-03-09T00:25:18.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2534999999999999e-05 2020-03-09T00:25:18.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823268+0000", 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2049999999999996e-06 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823276+0000", 2020-03-09T00:25:18.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3608999999999994e-05 2020-03-09T00:25:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823339+0000", 2020-03-09T00:25:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6861e-05 2020-03-09T00:25:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.823356+0000", 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021750749999999998 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825531+0000", 2020-03-09T00:25:18.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4615e-05 2020-03-09T00:25:18.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.825566+0000", 2020-03-09T00:25:18.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023877159999999998 2020-03-09T00:25:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s3 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.827819+0000", 2020-03-09T00:25:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.317617208000001, 2020-03-09T00:25:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029715000000000002, 2020-03-09T00:25:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.234 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.827819+0000", 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.827819+0000", 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000175255 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.827994+0000", 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6792999999999999e-05 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.828021+0000", 2020-03-09T00:25:18.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6384000000000001e-05 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.828037+0000", 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.695e-06 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.828045+0000", 2020-03-09T00:25:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1168999999999997e-05 2020-03-09T00:25:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.828096+0000", 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0369999999999995e-06 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.828104+0000", 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00264665 2020-03-09T00:25:18.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.830751+0000", 2020-03-09T00:25:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9517000000000001e-05 2020-03-09T00:25:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.830790+0000", 2020-03-09T00:25:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029715000000000002 2020-03-09T00:25:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s3 38/32 ECSubWrite(tid=11, reqid=client.4751.0:197, at_version=38'7, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.833070+0000", 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.312365991, 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021144089999999998, 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833070+0000", 2020-03-09T00:25:18.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833070+0000", 2020-03-09T00:25:18.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8577000000000003e-05 2020-03-09T00:25:18.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833148+0000", 2020-03-09T00:25:18.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5605999999999999e-05 2020-03-09T00:25:18.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833164+0000", 2020-03-09T00:25:18.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.234e-05 2020-03-09T00:25:18.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833176+0000", 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0900000000000004e-06 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833181+0000", 2020-03-09T00:25:18.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1678999999999998e-05 2020-03-09T00:25:18.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833233+0000", 2020-03-09T00:25:18.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2492e-05 2020-03-09T00:25:18.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.833246+0000", 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018892239999999999 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835135+0000", 2020-03-09T00:25:18.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9400999999999997e-05 2020-03-09T00:25:18.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835184+0000", 2020-03-09T00:25:18.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021144089999999998 2020-03-09T00:25:18.251 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.251 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.251 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:204 3.3s0 3:c7078095:::smithi10428818-50:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837426+0000", 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.308009589000001, 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063391309999999996, 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.252 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:18.253 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.253 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 204 2020-03-09T00:25:18.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.253 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837426+0000", 2020-03-09T00:25:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837426+0000", 2020-03-09T00:25:18.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010029100000000001 2020-03-09T00:25:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837527+0000", 2020-03-09T00:25:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0740000000000001e-05 2020-03-09T00:25:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837537+0000", 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5551000000000001e-05 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837553+0000", 2020-03-09T00:25:18.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1260999999999999e-05 2020-03-09T00:25:18.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837564+0000", 2020-03-09T00:25:18.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000126885 2020-03-09T00:25:18.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837691+0000", 2020-03-09T00:25:18.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000183003 2020-03-09T00:25:18.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837874+0000", 2020-03-09T00:25:18.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014477660000000001 2020-03-09T00:25:18.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.839322+0000", 2020-03-09T00:25:18.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002911411 2020-03-09T00:25:18.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842233+0000", 2020-03-09T00:25:18.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014388719999999999 2020-03-09T00:25:18.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:18.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843672+0000", 2020-03-09T00:25:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3350999999999994e-05 2020-03-09T00:25:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843765+0000", 2020-03-09T00:25:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063391309999999996 2020-03-09T00:25:18.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:205 3.3s0 3:c7078095:::smithi10428818-50:head [rollback 4] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837668+0000", 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.307767883, 2020-03-09T00:25:18.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075315549999999997, 2020-03-09T00:25:18.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.265 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:18.266 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.266 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 205 2020-03-09T00:25:18.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.266 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837668+0000", 2020-03-09T00:25:18.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837668+0000", 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7719999999999994e-05 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837746+0000", 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0366999999999999e-05 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837756+0000", 2020-03-09T00:25:18.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3539e-05 2020-03-09T00:25:18.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837770+0000", 2020-03-09T00:25:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9030000000000004e-06 2020-03-09T00:25:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837777+0000", 2020-03-09T00:25:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026689600000000002 2020-03-09T00:25:18.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840446+0000", 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013164399999999999 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840578+0000", 2020-03-09T00:25:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089358000000000005 2020-03-09T00:25:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841472+0000", 2020-03-09T00:25:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023155659999999998 2020-03-09T00:25:18.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843787+0000", 2020-03-09T00:25:18.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013292079999999999 2020-03-09T00:25:18.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845116+0000", 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3067999999999998e-05 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845199+0000", 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075315549999999997 2020-03-09T00:25:18.274 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.275 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.275 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.275 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:206 3.3s0 3:c7078095:::smithi10428818-50:head [append 0~0] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:18.275 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837837+0000", 2020-03-09T00:25:18.275 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.307598968000001, 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080301509999999993, 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 206 2020-03-09T00:25:18.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837837+0000", 2020-03-09T00:25:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837837+0000", 2020-03-09T00:25:18.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0633000000000001e-05 2020-03-09T00:25:18.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837887+0000", 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3880000000000002e-06 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837895+0000", 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7628e-05 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837922+0000", 2020-03-09T00:25:18.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3680000000000001e-06 2020-03-09T00:25:18.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837928+0000", 2020-03-09T00:25:18.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045821450000000001 2020-03-09T00:25:18.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842510+0000", 2020-03-09T00:25:18.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000126601 2020-03-09T00:25:18.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:18.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842637+0000", 2020-03-09T00:25:18.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065715299999999995 2020-03-09T00:25:18.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843294+0000", 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056297900000000002 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843857+0000", 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019226499999999999 2020-03-09T00:25:18.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:18.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845779+0000", 2020-03-09T00:25:18.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7606000000000001e-05 2020-03-09T00:25:18.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845867+0000", 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080301509999999993 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.285 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/32 ECSubWriteReply(tid=36, last_complete=38'11, committed=1, applied=1))", 2020-03-09T00:25:18.286 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.840088+0000", 2020-03-09T00:25:18.286 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.305347965999999, 2020-03-09T00:25:18.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021467869999999998, 2020-03-09T00:25:18.286 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.286 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.286 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840088+0000", 2020-03-09T00:25:18.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840088+0000", 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1094999999999997e-05 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840149+0000", 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5680000000000005e-06 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840159+0000", 2020-03-09T00:25:18.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4572e-05 2020-03-09T00:25:18.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840173+0000", 2020-03-09T00:25:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5539999999999998e-06 2020-03-09T00:25:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840182+0000", 2020-03-09T00:25:18.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020326789999999999 2020-03-09T00:25:18.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842214+0000", 2020-03-09T00:25:18.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0319e-05 2020-03-09T00:25:18.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842235+0000", 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021467869999999998 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 38/32 ECSubWriteReply(tid=36, last_complete=38'11, committed=1, applied=1))", 2020-03-09T00:25:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.840770+0000", 2020-03-09T00:25:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.304666152999999, 2020-03-09T00:25:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001509441, 2020-03-09T00:25:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.293 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840770+0000", 2020-03-09T00:25:18.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840770+0000", 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.18e-05 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840801+0000", 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.767e-06 2020-03-09T00:25:18.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840807+0000", 2020-03-09T00:25:18.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7620000000000001e-06 2020-03-09T00:25:18.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840816+0000", 2020-03-09T00:25:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9270000000000005e-06 2020-03-09T00:25:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.840824+0000", 2020-03-09T00:25:18.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001442481 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842266+0000", 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2704000000000001e-05 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842279+0000", 2020-03-09T00:25:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001509441 2020-03-09T00:25:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s3 42/32 ECSubWrite(tid=21, reqid=osd.15.0:21, at_version=42'14, trim_to=0'0, roll_forward_to=40'12))", 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:51.853183+0000", 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.292253164000002, 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002442367, 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853183+0000", 2020-03-09T00:25:18.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853183+0000", 2020-03-09T00:25:18.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5548000000000002e-05 2020-03-09T00:25:18.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853258+0000", 2020-03-09T00:25:18.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0499999999999999e-05 2020-03-09T00:25:18.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853289+0000", 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3003999999999999e-05 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853312+0000", 2020-03-09T00:25:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2269999999999999e-05 2020-03-09T00:25:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853324+0000", 2020-03-09T00:25:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2193000000000006e-05 2020-03-09T00:25:18.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853386+0000", 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3115000000000001e-05 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.853399+0000", 2020-03-09T00:25:18.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002164558 2020-03-09T00:25:18.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.855564+0000", 2020-03-09T00:25:18.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1179000000000005e-05 2020-03-09T00:25:18.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:51.855625+0000", 2020-03-09T00:25:18.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002442367 2020-03-09T00:25:18.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.307 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 44/32 ECSubWrite(tid=52, reqid=client.4751.0:246, at_version=44'19, trim_to=0'0, roll_forward_to=44'17))", 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.854833+0000", 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.290602363000001, 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001866943, 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.308 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854833+0000", 2020-03-09T00:25:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854833+0000", 2020-03-09T00:25:18.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3008000000000004e-05 2020-03-09T00:25:18.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854896+0000", 2020-03-09T00:25:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7529999999999999e-05 2020-03-09T00:25:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854924+0000", 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3495000000000001e-05 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854947+0000", 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2039999999999997e-06 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854956+0000", 2020-03-09T00:25:18.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2422999999999995e-05 2020-03-09T00:25:18.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855018+0000", 2020-03-09T00:25:18.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4572999999999999e-05 2020-03-09T00:25:18.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855033+0000", 2020-03-09T00:25:18.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016165999999999999 2020-03-09T00:25:18.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856649+0000", 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1109999999999997e-05 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856700+0000", 2020-03-09T00:25:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001866943 2020-03-09T00:25:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.316 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 44/32 ECSubWrite(tid=53, reqid=client.4751.0:247, at_version=44'20, trim_to=0'0, roll_forward_to=44'17))", 2020-03-09T00:25:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.855017+0000", 2020-03-09T00:25:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.290418950999999, 2020-03-09T00:25:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001723708, 2020-03-09T00:25:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855017+0000", 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855017+0000", 2020-03-09T00:25:18.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6921000000000005e-05 2020-03-09T00:25:18.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855084+0000", 2020-03-09T00:25:18.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1798e-05 2020-03-09T00:25:18.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855106+0000", 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6076999999999999e-05 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855122+0000", 2020-03-09T00:25:18.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8300000000000001e-06 2020-03-09T00:25:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855127+0000", 2020-03-09T00:25:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074394200000000004 2020-03-09T00:25:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855871+0000", 2020-03-09T00:25:18.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3611999999999999e-05 2020-03-09T00:25:18.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855885+0000", 2020-03-09T00:25:18.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081994200000000004 2020-03-09T00:25:18.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856705+0000", 2020-03-09T00:25:18.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5586000000000001e-05 2020-03-09T00:25:18.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856741+0000", 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001723708 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.325 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 44/32 ECSubWrite(tid=54, reqid=client.4751.0:248, at_version=44'21, trim_to=0'0, roll_forward_to=44'17))", 2020-03-09T00:25:18.325 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.855190+0000", 2020-03-09T00:25:18.325 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.290246217, 2020-03-09T00:25:18.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001713472, 2020-03-09T00:25:18.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.325 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.326 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855190+0000", 2020-03-09T00:25:18.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855190+0000", 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6105000000000001e-05 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855226+0000", 2020-03-09T00:25:18.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.324e-05 2020-03-09T00:25:18.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855239+0000", 2020-03-09T00:25:18.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1358e-05 2020-03-09T00:25:18.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855250+0000", 2020-03-09T00:25:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7810000000000002e-06 2020-03-09T00:25:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.855256+0000", 2020-03-09T00:25:18.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011822099999999999 2020-03-09T00:25:18.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856438+0000", 2020-03-09T00:25:18.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2884999999999999e-05 2020-03-09T00:25:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856451+0000", 2020-03-09T00:25:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038472199999999999 2020-03-09T00:25:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856836+0000", 2020-03-09T00:25:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7170999999999998e-05 2020-03-09T00:25:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856903+0000", 2020-03-09T00:25:18.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001713472 2020-03-09T00:25:18.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.333 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 44/32 ECSubWrite(tid=56, reqid=client.4751.0:254, at_version=44'23, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:25:18.334 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863925+0000", 2020-03-09T00:25:18.334 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.281511242000001, 2020-03-09T00:25:18.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019154630000000001, 2020-03-09T00:25:18.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:18.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.334 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:18.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:18.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863925+0000", 2020-03-09T00:25:18.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:18.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:18.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863925+0000", 2020-03-09T00:25:18.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014875500000000001 2020-03-09T00:25:18.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:18.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864073+0000", 2020-03-09T00:25:18.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6265999999999999e-05 2020-03-09T00:25:18.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:18.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864100+0000", 2020-03-09T00:25:18.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5849999999999999e-05 2020-03-09T00:25:18.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864115+0000", 2020-03-09T00:25:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3430000000000002e-06 2020-03-09T00:25:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864125+0000", 2020-03-09T00:25:18.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6619000000000001e-05 2020-03-09T00:25:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864181+0000", 2020-03-09T00:25:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3569e-05 2020-03-09T00:25:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864195+0000", 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00156913 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865764+0000", 2020-03-09T00:25:18.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5931000000000002e-05 2020-03-09T00:25:18.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:18.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:18.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:18.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865840+0000", 2020-03-09T00:25:18.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019154630000000001 2020-03-09T00:25:18.341 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.341 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:18.342 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:18.342 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:18.343 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:18.341+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:18.344 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:25:18.444 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:18.442+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:18.460 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:18.461 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:18.469 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:25:18.545 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:18.543+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:18.585 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:18.585 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:18.585 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:18.585 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:18.586 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:18.594 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:25:18.645 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:18.643+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:18.712 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:18.712 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:18.712 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:18.713 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:18.713 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.713 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:251 3.0s0 3:0efbb6fa:::smithi10428818-44:head [copy-from ver 5] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:18.713 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:18.713 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.860190225, 2020-03-09T00:25:18.713 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001, 2020-03-09T00:25:18.713 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 251 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.715 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.715 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:18.715 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.715 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.715 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.715 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:18.716 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6965999999999995e-05 2020-03-09T00:25:18.716 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.716 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.716 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.716 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849062+0000", 2020-03-09T00:25:18.716 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6689999999999995e-06 2020-03-09T00:25:18.716 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849071+0000", 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7941999999999999e-05 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.717 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849089+0000", 2020-03-09T00:25:18.718 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9480000000000003e-06 2020-03-09T00:25:18.718 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.718 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.718 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.718 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849099+0000", 2020-03-09T00:25:18.718 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4438000000000003e-05 2020-03-09T00:25:18.718 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.719 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.719 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.719 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849153+0000", 2020-03-09T00:25:18.719 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012488000000000001 2020-03-09T00:25:18.719 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.719 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.719 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.720 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849278+0000", 2020-03-09T00:25:18.720 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013412429999999999 2020-03-09T00:25:18.720 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.720 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.720 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.720 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862691+0000", 2020-03-09T00:25:18.720 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011733578 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.874424+0000", 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000863506 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.721 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.722 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875288+0000", 2020-03-09T00:25:18.722 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9488999999999998e-05 2020-03-09T00:25:18.722 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.722 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.722 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.722 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875357+0000", 2020-03-09T00:25:18.722 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001 2020-03-09T00:25:18.723 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.723 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.723 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.723 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.723 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.723 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:252 3.0s0 3:0efbb6fa:::smithi10428818-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:18.723 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:18.724 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.860041426999999, 2020-03-09T00:25:18.724 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368, 2020-03-09T00:25:18.724 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.724 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:18.724 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.724 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.724 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.725 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 252 2020-03-09T00:25:18.725 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.725 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.725 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.725 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.725 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:18.725 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.726 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.726 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.726 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.726 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:18.726 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7280999999999998e-05 2020-03-09T00:25:18.726 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.726 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.727 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.727 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849171+0000", 2020-03-09T00:25:18.727 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6030000000000001e-06 2020-03-09T00:25:18.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.727 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.727 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.727 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849178+0000", 2020-03-09T00:25:18.728 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.308e-05 2020-03-09T00:25:18.728 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.728 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.728 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.728 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849191+0000", 2020-03-09T00:25:18.728 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.553e-06 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849197+0000", 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030484499999999999 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.729 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.730 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849501+0000", 2020-03-09T00:25:18.730 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5923e-05 2020-03-09T00:25:18.730 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.730 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.730 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:18.730 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849557+0000", 2020-03-09T00:25:18.730 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019572935999999999 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869130+0000", 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013702000000000001 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.731 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:18.732 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869267+0000", 2020-03-09T00:25:18.732 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0068816229999999999 2020-03-09T00:25:18.732 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.732 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.732 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.732 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876149+0000", 2020-03-09T00:25:18.732 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8204999999999999e-05 2020-03-09T00:25:18.733 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.733 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.733 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.733 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876247+0000", 2020-03-09T00:25:18.733 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109299 2020-03-09T00:25:18.733 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.734 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.734 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.734 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876356+0000", 2020-03-09T00:25:18.734 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368 2020-03-09T00:25:18.734 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.734 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.734 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.735 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.735 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.735 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs1 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:18.735 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:18.735 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.858085781, 2020-03-09T00:25:18.735 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026627181999999999, 2020-03-09T00:25:18.735 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.736 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.737 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.737 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:18.737 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058834270000000001 2020-03-09T00:25:18.737 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.737 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.737 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.737 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.856973+0000", 2020-03-09T00:25:18.738 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1265999999999999e-05 2020-03-09T00:25:18.738 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.738 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.738 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.738 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.856994+0000", 2020-03-09T00:25:18.738 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5070000000000006e-06 2020-03-09T00:25:18.739 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.739 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.739 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.739 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857004+0000", 2020-03-09T00:25:18.739 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0560000000000004e-06 2020-03-09T00:25:18.739 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.739 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857009+0000", 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1946e-05 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857061+0000", 2020-03-09T00:25:18.740 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4282e-05 2020-03-09T00:25:18.741 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.741 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.741 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.741 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857075+0000", 2020-03-09T00:25:18.741 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020582362999999999 2020-03-09T00:25:18.741 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.741 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.742 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.742 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.877658+0000", 2020-03-09T00:25:18.742 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9335e-05 2020-03-09T00:25:18.742 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.742 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.742 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.742 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.877717+0000", 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026627181999999999 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:412 3.0s0 3:054f30c0:::smithi10428818-49:head [copy-from ver 0] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:18.743 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:18.744 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 14.865664220999999, 2020-03-09T00:25:18.744 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.97896074700000002, 2020-03-09T00:25:18.744 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.744 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:18.744 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.744 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.744 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.745 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 412 2020-03-09T00:25:18.745 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.745 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.745 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.745 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.745 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:18.745 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014666000000000001 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.746 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843658+0000", 2020-03-09T00:25:18.747 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0059e-05 2020-03-09T00:25:18.747 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.747 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843668+0000", 2020-03-09T00:25:18.747 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8175000000000001e-05 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843686+0000", 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0209999999999999e-06 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.748 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.749 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843694+0000", 2020-03-09T00:25:18.749 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8402000000000001e-05 2020-03-09T00:25:18.749 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.749 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.749 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.749 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843753+0000", 2020-03-09T00:25:18.749 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9160000000000001e-06 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.95898846199999999 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.750 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.751 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.802744+0000", 2020-03-09T00:25:18.751 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019997600000000001 2020-03-09T00:25:18.751 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.751 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.751 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.751 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.802944+0000", 2020-03-09T00:25:18.751 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019528075999999998 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.822472+0000", 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.97896074700000002 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.752 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.753 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.753 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.753 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:413 3.0s0 3:054f30c0:::smithi10428818-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:18.753 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:18.753 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 14.865420691000001, 2020-03-09T00:25:18.753 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.98098931099999997, 2020-03-09T00:25:18.753 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 413 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.754 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.755 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.755 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:18.755 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.755 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.755 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.755 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.755 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4046999999999999e-05 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843809+0000", 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1140000000000006e-06 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.757 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.757 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843817+0000", 2020-03-09T00:25:18.757 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5950000000000001e-05 2020-03-09T00:25:18.757 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.757 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.757 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.757 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843833+0000", 2020-03-09T00:25:18.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4799999999999998e-06 2020-03-09T00:25:18.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.758 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.758 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843839+0000", 2020-03-09T00:25:18.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6656000000000003e-05 2020-03-09T00:25:18.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.759 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.759 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843876+0000", 2020-03-09T00:25:18.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8109999999999999e-06 2020-03-09T00:25:18.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843878+0000", 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.95925583699999994 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803134+0000", 2020-03-09T00:25:18.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1947999999999997e-05 2020-03-09T00:25:18.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.761 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:18.761 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803196+0000", 2020-03-09T00:25:18.761 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020717148000000001 2020-03-09T00:25:18.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.762 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.823913+0000", 2020-03-09T00:25:18.762 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012539800000000001 2020-03-09T00:25:18.762 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.762 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.762 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.824038+0000", 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00070592199999999999 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.824744+0000", 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.98098931099999997 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.763 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:420 3.0s0 3:07071236:::smithi10428818-3:head [rollback 10] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 14.503940303, 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61566089000000002, 2020-03-09T00:25:18.764 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.765 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.765 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.765 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.765 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.765 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 420 2020-03-09T00:25:18.765 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.765 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.766 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:18.766 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.766 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5570000000000001e-05 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205261+0000", 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8359999999999997e-06 2020-03-09T00:25:18.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.768 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.768 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205267+0000", 2020-03-09T00:25:18.768 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0376e-05 2020-03-09T00:25:18.768 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.768 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205277+0000", 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4229999999999999e-06 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205284+0000", 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2729000000000003e-05 2020-03-09T00:25:18.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205337+0000", 2020-03-09T00:25:18.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8240000000000001e-06 2020-03-09T00:25:18.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:18.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205341+0000", 2020-03-09T00:25:18.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.59787851299999994 2020-03-09T00:25:18.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.771 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803219+0000", 2020-03-09T00:25:18.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000158166 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803378+0000", 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012380760000000001 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804616+0000", 2020-03-09T00:25:18.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037069310000000001 2020-03-09T00:25:18.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.808323+0000", 2020-03-09T00:25:18.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012483437999999999 2020-03-09T00:25:18.774 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820806+0000", 2020-03-09T00:25:18.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0007999999999996e-05 2020-03-09T00:25:18.774 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820896+0000", 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61566089000000002 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:421 3.0s0 3:07071236:::smithi10428818-3:head [append 0~0] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 14.503842825, 2020-03-09T00:25:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61577959800000004, 2020-03-09T00:25:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 421 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:18.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:18.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1856000000000001e-05 2020-03-09T00:25:18.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205364+0000", 2020-03-09T00:25:18.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9029999999999997e-06 2020-03-09T00:25:18.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205368+0000", 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2529999999999998e-06 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205376+0000", 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.004e-06 2020-03-09T00:25:18.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205379+0000", 2020-03-09T00:25:18.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3745999999999998e-05 2020-03-09T00:25:18.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205412+0000", 2020-03-09T00:25:18.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1799999999999999e-06 2020-03-09T00:25:18.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:18.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205414+0000", 2020-03-09T00:25:18.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.59956388100000002 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804977+0000", 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115327 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.783 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805093+0000", 2020-03-09T00:25:18.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00070765400000000005 2020-03-09T00:25:18.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805800+0000", 2020-03-09T00:25:18.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025236199999999999 2020-03-09T00:25:18.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.808324+0000", 2020-03-09T00:25:18.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012705651 2020-03-09T00:25:18.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.821030+0000", 2020-03-09T00:25:18.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2522999999999996e-05 2020-03-09T00:25:18.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.821112+0000", 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61577959800000004 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.788 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:488 3.9s0 3:93e8184b:::smithi10428818-7:head [copy-from ver 2] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:18.788 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:18.788 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8974357679999994, 2020-03-09T00:25:18.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033520886, 2020-03-09T00:25:18.788 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.788 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.788 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 488 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:18.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.790 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:18.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7763999999999996e-05 2020-03-09T00:25:18.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811817+0000", 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.093e-05 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811828+0000", 2020-03-09T00:25:18.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6980999999999999e-05 2020-03-09T00:25:18.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811845+0000", 2020-03-09T00:25:18.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9039999999999998e-06 2020-03-09T00:25:18.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.793 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811854+0000", 2020-03-09T00:25:18.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079268800000000005 2020-03-09T00:25:18.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.794 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812647+0000", 2020-03-09T00:25:18.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.132e-05 2020-03-09T00:25:18.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:18.795 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812658+0000", 2020-03-09T00:25:18.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043188640000000004 2020-03-09T00:25:18.795 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.795 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.795 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.795 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.816977+0000", 2020-03-09T00:25:18.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109483 2020-03-09T00:25:18.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817087+0000", 2020-03-09T00:25:18.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011389424 2020-03-09T00:25:18.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.828476+0000", 2020-03-09T00:25:18.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048806689999999998 2020-03-09T00:25:18.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833357+0000", 2020-03-09T00:25:18.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011728186999999999 2020-03-09T00:25:18.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845085+0000", 2020-03-09T00:25:18.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017567200000000001 2020-03-09T00:25:18.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845261+0000", 2020-03-09T00:25:18.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033520886 2020-03-09T00:25:18.800 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.800 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.801 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.801 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:489 3.9s0 3:93e8184b:::smithi10428818-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8972669960000008, 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037452473999999999, 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.802 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.803 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 489 2020-03-09T00:25:18.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.803 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:18.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:18.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2522999999999999e-05 2020-03-09T00:25:18.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.805 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.805 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811951+0000", 2020-03-09T00:25:18.805 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9319999999999999e-06 2020-03-09T00:25:18.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.806 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811958+0000", 2020-03-09T00:25:18.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3498e-05 2020-03-09T00:25:18.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.806 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811971+0000", 2020-03-09T00:25:18.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7749999999999998e-06 2020-03-09T00:25:18.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.807 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811977+0000", 2020-03-09T00:25:18.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00075333400000000003 2020-03-09T00:25:18.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812731+0000", 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.183e-06 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:18.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812740+0000", 2020-03-09T00:25:18.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005894598 2020-03-09T00:25:18.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818634+0000", 2020-03-09T00:25:18.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4839000000000006e-05 2020-03-09T00:25:18.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:18.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818709+0000", 2020-03-09T00:25:18.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010967121999999999 2020-03-09T00:25:18.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829676+0000", 2020-03-09T00:25:18.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013027000000000001 2020-03-09T00:25:18.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:18.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829807+0000", 2020-03-09T00:25:18.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019337825999999999 2020-03-09T00:25:18.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849144+0000", 2020-03-09T00:25:18.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012293699999999999 2020-03-09T00:25:18.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.813 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.813 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849267+0000", 2020-03-09T00:25:18.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3636999999999998e-05 2020-03-09T00:25:18.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.813 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.814 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849361+0000", 2020-03-09T00:25:18.814 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037452473999999999 2020-03-09T00:25:18.814 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.814 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.814 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.815 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:490 3.0s0 3:07071236:::smithi10428818-3:head [append 0~786432] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:18.815 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:18.815 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8902098790000004, 2020-03-09T00:25:18.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049768433000000001, 2020-03-09T00:25:18.815 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.815 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.815 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.816 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.816 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.816 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 490 2020-03-09T00:25:18.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.816 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.816 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:18.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:18.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040424099999999998 2020-03-09T00:25:18.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823008+0000", 2020-03-09T00:25:18.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8820000000000007e-06 2020-03-09T00:25:18.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823017+0000", 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6317e-05 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823033+0000", 2020-03-09T00:25:18.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.622e-06 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823041+0000", 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034770550000000002 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.826518+0000", 2020-03-09T00:25:18.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074775010000000001 2020-03-09T00:25:18.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833995+0000", 2020-03-09T00:25:18.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017771289999999999 2020-03-09T00:25:18.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.835772+0000", 2020-03-09T00:25:18.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0086191449999999999 2020-03-09T00:25:18.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844392+0000", 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024243164000000001 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.868635+0000", 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9208000000000003e-05 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.825 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.825 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.868734+0000", 2020-03-09T00:25:18.825 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049768433000000001 2020-03-09T00:25:18.826 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.826 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.826 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.826 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:491 3.0s0 3:07071236:::smithi10428818-3:head [append 0~491520] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:18.826 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:18.827 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8860747189999998, 2020-03-09T00:25:18.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046647411, 2020-03-09T00:25:18.827 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.827 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.827 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.827 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.827 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 491 2020-03-09T00:25:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:18.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00087253199999999997 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823973+0000", 2020-03-09T00:25:18.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8790000000000003e-06 2020-03-09T00:25:18.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823981+0000", 2020-03-09T00:25:18.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5770999999999999e-05 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823997+0000", 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9669999999999998e-06 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824003+0000", 2020-03-09T00:25:18.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021062780999999999 2020-03-09T00:25:18.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845066+0000", 2020-03-09T00:25:18.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8070000000000002e-05 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845154+0000", 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00088644099999999997 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.846040+0000", 2020-03-09T00:25:18.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084721699999999994 2020-03-09T00:25:18.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854512+0000", 2020-03-09T00:25:18.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015183247 2020-03-09T00:25:18.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869696+0000", 2020-03-09T00:25:18.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2553000000000003e-05 2020-03-09T00:25:18.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869748+0000", 2020-03-09T00:25:18.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046647411 2020-03-09T00:25:18.837 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:492 3.0s0 3:07071236:::smithi10428818-3:head [append 0~278528] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8851200200000005, 2020-03-09T00:25:18.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045823801999999997, 2020-03-09T00:25:18.839 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.839 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.839 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.839 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.839 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.839 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 492 2020-03-09T00:25:18.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.840 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.840 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:18.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044278900000000002 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824498+0000", 2020-03-09T00:25:18.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7820000000000005e-06 2020-03-09T00:25:18.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824508+0000", 2020-03-09T00:25:18.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3557000000000001e-05 2020-03-09T00:25:18.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824522+0000", 2020-03-09T00:25:18.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7799999999999997e-06 2020-03-09T00:25:18.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824527+0000", 2020-03-09T00:25:18.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024667254 2020-03-09T00:25:18.844 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849195+0000", 2020-03-09T00:25:18.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133366 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849328+0000", 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001009271 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.846 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.850337+0000", 2020-03-09T00:25:18.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016518857000000001 2020-03-09T00:25:18.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.866856+0000", 2020-03-09T00:25:18.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002975601 2020-03-09T00:25:18.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.847 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869832+0000", 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7545e-05 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869879+0000", 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045823801999999997 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.848 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.849 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.849 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.849 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.849 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:493 3.0s0 3:07071236:::smithi10428818-3:head [setxattr _header (56) in=7b] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:18.849 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:18.849 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8845982760000002, 2020-03-09T00:25:18.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045571699, 2020-03-09T00:25:18.850 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.850 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.850 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 493 2020-03-09T00:25:18.850 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.851 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:18.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.851 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1864000000000001e-05 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824659+0000", 2020-03-09T00:25:18.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1690000000000003e-06 2020-03-09T00:25:18.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.853 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.853 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824666+0000", 2020-03-09T00:25:18.853 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3011e-05 2020-03-09T00:25:18.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824679+0000", 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6269999999999997e-06 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824685+0000", 2020-03-09T00:25:18.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.029865282 2020-03-09T00:25:18.855 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.855 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.855 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854550+0000", 2020-03-09T00:25:18.855 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015000699999999999 2020-03-09T00:25:18.855 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.856 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.856 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854700+0000", 2020-03-09T00:25:18.856 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032691450000000002 2020-03-09T00:25:18.856 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.856 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.856 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.857 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.857969+0000", 2020-03-09T00:25:18.857 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088895759999999997 2020-03-09T00:25:18.857 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.857 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.857 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.866859+0000", 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032396439999999999 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870099+0000", 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0374000000000003e-05 2020-03-09T00:25:18.858 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.859 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.859 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.859 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870149+0000", 2020-03-09T00:25:18.859 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045571699 2020-03-09T00:25:18.859 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.859 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.859 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.860 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.860 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.860 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:494 3.0s0 3:07071236:::smithi10428818-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:18.860 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:18.860 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8844475349999996, 2020-03-09T00:25:18.860 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048769923, 2020-03-09T00:25:18.860 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.861 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:18.861 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.861 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.861 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.861 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 494 2020-03-09T00:25:18.861 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.862 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.862 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.862 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.862 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:18.862 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.862 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.863 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.863 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.863 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:18.863 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1047999999999997e-05 2020-03-09T00:25:18.863 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.863 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.863 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824769+0000", 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5699999999999998e-06 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824776+0000", 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2532e-05 2020-03-09T00:25:18.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.865 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.865 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.865 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824788+0000", 2020-03-09T00:25:18.865 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0990000000000001e-06 2020-03-09T00:25:18.865 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.865 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.865 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.866 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824792+0000", 2020-03-09T00:25:18.866 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035212312000000003 2020-03-09T00:25:18.866 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.866 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.866 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860004+0000", 2020-03-09T00:25:18.866 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000129932 2020-03-09T00:25:18.867 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.867 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.867 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:18.868 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860134+0000", 2020-03-09T00:25:18.868 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010193544000000001 2020-03-09T00:25:18.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.869 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.869 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870328+0000", 2020-03-09T00:25:18.869 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3980999999999995e-05 2020-03-09T00:25:18.869 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.869 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870402+0000", 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030959049999999999 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.873498+0000", 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048769923 2020-03-09T00:25:18.870 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.871 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.871 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.871 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:506 3.0s0 3:05d2883d:::smithi10428818-38:head [stat,delete] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:18.871 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:18.871 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.8749618550000005, 2020-03-09T00:25:18.872 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037224634, 2020-03-09T00:25:18.872 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.872 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.872 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.872 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.872 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.872 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 506 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8146999999999998e-05 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834262+0000", 2020-03-09T00:25:18.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4239999999999993e-06 2020-03-09T00:25:18.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.875 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.875 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.875 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834270+0000", 2020-03-09T00:25:18.875 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5318000000000001e-05 2020-03-09T00:25:18.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.875 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.876 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.876 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834285+0000", 2020-03-09T00:25:18.876 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5679999999999996e-06 2020-03-09T00:25:18.876 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.876 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.876 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.876 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834292+0000", 2020-03-09T00:25:18.877 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025873846999999998 2020-03-09T00:25:18.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.877 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.877 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.877 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860166+0000", 2020-03-09T00:25:18.877 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017915800000000001 2020-03-09T00:25:18.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.878 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.878 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.878 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860345+0000", 2020-03-09T00:25:18.878 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040042910000000001 2020-03-09T00:25:18.878 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.878 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.878 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.879 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.864349+0000", 2020-03-09T00:25:18.879 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029197120000000001 2020-03-09T00:25:18.879 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.879 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.880 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.880 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.867269+0000", 2020-03-09T00:25:18.881 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0041192809999999998 2020-03-09T00:25:18.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.881 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.881 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.871388+0000", 2020-03-09T00:25:18.881 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9888000000000001e-05 2020-03-09T00:25:18.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.882 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.882 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.871438+0000", 2020-03-09T00:25:18.882 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037224634 2020-03-09T00:25:18.882 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.882 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.883 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.883 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.883 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:530 3.9s0 3:93e8184b:::smithi10428818-7:head [append 0~475136] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:18.883 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.774914+0000", 2020-03-09T00:25:18.883 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.9342614420000004, 2020-03-09T00:25:18.883 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025051856000000001, 2020-03-09T00:25:18.884 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.884 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.884 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.884 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.884 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.884 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 530 2020-03-09T00:25:18.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.885 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.885 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.885 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.885 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.774914+0000", 2020-03-09T00:25:18.885 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.885 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.886 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.886 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.774914+0000", 2020-03-09T00:25:18.886 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001782779 2020-03-09T00:25:18.886 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776697+0000", 2020-03-09T00:25:18.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2039999999999997e-06 2020-03-09T00:25:18.887 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776705+0000", 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4129e-05 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776719+0000", 2020-03-09T00:25:18.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3509999999999996e-06 2020-03-09T00:25:18.889 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.889 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.889 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.889 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776726+0000", 2020-03-09T00:25:18.889 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6595999999999997e-05 2020-03-09T00:25:18.889 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.890 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.890 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.890 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776793+0000", 2020-03-09T00:25:18.890 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3156000000000001e-05 2020-03-09T00:25:18.890 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.890 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.890 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:18.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776806+0000", 2020-03-09T00:25:18.891 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042757180000000004 2020-03-09T00:25:18.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781082+0000", 2020-03-09T00:25:18.891 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013743100000000001 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781219+0000", 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015642130000000001 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.782784+0000", 2020-03-09T00:25:18.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088951740000000005 2020-03-09T00:25:18.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.893 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.791679+0000", 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0079903809999999995 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.799669+0000", 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000296724 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.799966+0000", 2020-03-09T00:25:18.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025051856000000001 2020-03-09T00:25:18.895 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.895 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.895 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.896 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:532 3.9s0 3:93e8184b:::smithi10428818-7:head [append 0~442368] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:18.896 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.781309+0000", 2020-03-09T00:25:18.896 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.9278668899999998, 2020-03-09T00:25:18.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024957796000000001, 2020-03-09T00:25:18.896 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.896 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.897 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 532 2020-03-09T00:25:18.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.897 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781309+0000", 2020-03-09T00:25:18.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781309+0000", 2020-03-09T00:25:18.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050756800000000002 2020-03-09T00:25:18.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781816+0000", 2020-03-09T00:25:18.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3939999999999996e-06 2020-03-09T00:25:18.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781825+0000", 2020-03-09T00:25:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4737e-05 2020-03-09T00:25:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781839+0000", 2020-03-09T00:25:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2359999999999998e-06 2020-03-09T00:25:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781847+0000", 2020-03-09T00:25:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01001485 2020-03-09T00:25:18.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.791861+0000", 2020-03-09T00:25:18.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000134719 2020-03-09T00:25:18.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.791996+0000", 2020-03-09T00:25:18.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00070887000000000001 2020-03-09T00:25:18.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.792705+0000", 2020-03-09T00:25:18.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0073357869999999999 2020-03-09T00:25:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.800041+0000", 2020-03-09T00:25:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0061292969999999997 2020-03-09T00:25:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.806170+0000", 2020-03-09T00:25:18.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6337999999999998e-05 2020-03-09T00:25:18.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.806266+0000", 2020-03-09T00:25:18.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024957796000000001 2020-03-09T00:25:18.909 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.909 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.909 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.910 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:535 3.9s0 3:93e8184b:::smithi10428818-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e64)", 2020-03-09T00:25:18.910 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:18.910 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.9254387880000001, 2020-03-09T00:25:18.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026643350999999999, 2020-03-09T00:25:18.910 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.910 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:18.911 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.911 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.911 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.911 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 535 2020-03-09T00:25:18.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.911 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:18.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5562999999999997e-05 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783782+0000", 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5560000000000004e-06 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783789+0000", 2020-03-09T00:25:18.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2455e-05 2020-03-09T00:25:18.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783801+0000", 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.386e-06 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783806+0000", 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018586522000000001 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.802392+0000", 2020-03-09T00:25:18.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7225999999999995e-05 2020-03-09T00:25:18.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:18.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.802479+0000", 2020-03-09T00:25:18.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051016660000000004 2020-03-09T00:25:18.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.807581+0000", 2020-03-09T00:25:18.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6997999999999998e-05 2020-03-09T00:25:18.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.807648+0000", 2020-03-09T00:25:18.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027319789999999998 2020-03-09T00:25:18.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.810380+0000", 2020-03-09T00:25:18.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026643350999999999 2020-03-09T00:25:18.919 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.919 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.920 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.920 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:542 3.9s0 3:9f11c47d:::smithi10428818-37:head [rollback c] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:18.920 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.783928+0000", 2020-03-09T00:25:18.920 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.9252474729999998, 2020-03-09T00:25:18.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025038801999999999, 2020-03-09T00:25:18.921 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.921 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.921 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.921 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.921 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.921 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 542 2020-03-09T00:25:18.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.922 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.922 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783928+0000", 2020-03-09T00:25:18.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783928+0000", 2020-03-09T00:25:18.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5483e-05 2020-03-09T00:25:18.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783953+0000", 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4010000000000001e-06 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783957+0000", 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7279999999999995e-06 2020-03-09T00:25:18.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783965+0000", 2020-03-09T00:25:18.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.561e-06 2020-03-09T00:25:18.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783969+0000", 2020-03-09T00:25:18.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019562218999999999 2020-03-09T00:25:18.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.803531+0000", 2020-03-09T00:25:18.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012549699999999999 2020-03-09T00:25:18.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.803657+0000", 2020-03-09T00:25:18.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00054354800000000003 2020-03-09T00:25:18.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.804200+0000", 2020-03-09T00:25:18.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023566770000000002 2020-03-09T00:25:18.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.806557+0000", 2020-03-09T00:25:18.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002348003 2020-03-09T00:25:18.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.808905+0000", 2020-03-09T00:25:18.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1685000000000003e-05 2020-03-09T00:25:18.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.808967+0000", 2020-03-09T00:25:18.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025038801999999999 2020-03-09T00:25:18.931 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.931 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.931 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.931 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:543 3.9s0 3:9f11c47d:::smithi10428818-37:head [append 0~0] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:18.931 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.784011+0000", 2020-03-09T00:25:18.932 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 4.925164455, 2020-03-09T00:25:18.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025124014, 2020-03-09T00:25:18.932 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:18.932 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:18.932 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:18.932 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:18.932 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 543 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784011+0000", 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:18.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:18.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784011+0000", 2020-03-09T00:25:18.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0240000000000001e-05 2020-03-09T00:25:18.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:18.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784051+0000", 2020-03-09T00:25:18.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3740000000000003e-06 2020-03-09T00:25:18.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:18.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784058+0000", 2020-03-09T00:25:18.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.152e-05 2020-03-09T00:25:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784069+0000", 2020-03-09T00:25:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6880000000000002e-06 2020-03-09T00:25:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784076+0000", 2020-03-09T00:25:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020748419000000001 2020-03-09T00:25:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.804824+0000", 2020-03-09T00:25:18.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115017 2020-03-09T00:25:18.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:18.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.804939+0000", 2020-03-09T00:25:18.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00071181800000000002 2020-03-09T00:25:18.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:18.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.805651+0000", 2020-03-09T00:25:18.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00090687999999999999 2020-03-09T00:25:18.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:18.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.806558+0000", 2020-03-09T00:25:18.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025351779999999999 2020-03-09T00:25:18.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:18.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.809093+0000", 2020-03-09T00:25:18.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1879999999999999e-05 2020-03-09T00:25:18.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:18.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:18.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:18.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.809135+0000", 2020-03-09T00:25:18.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025124014 2020-03-09T00:25:18.942 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.942 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.942 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.942 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:18.942 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:18.943 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:18.949 INFO:teuthology.orchestra.run.smithi139.stderr:Exporting 2.7 info 2.7( empty local-lis/les=45/46 n=0 ec=27/27 lis/c=45/45 les/c/f=46/46/0 sis=45) 2020-03-09T00:25:18.949 INFO:teuthology.orchestra.run.smithi139.stderr:Export successful 2020-03-09T00:25:18.949 INFO:tasks.rados.rados.0.smithi104.stdout:231: done (8 left) 2020-03-09T00:25:18.950 INFO:tasks.rados.rados.0.smithi104.stdout:232: rollback oid 28 current snap is 17 2020-03-09T00:25:18.950 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 28 to 14 2020-03-09T00:25:18.950 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 4 v 46 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-03-09T00:25:18.950 INFO:tasks.rados.rados.0.smithi104.stdout:221: finishing write tid 2 to smithi10428818-23 2020-03-09T00:25:18.950 INFO:tasks.rados.rados.0.smithi104.stdout:221: finishing write tid 3 to smithi10428818-23 2020-03-09T00:25:18.951 INFO:tasks.rados.rados.0.smithi104.stdout:221: finishing write tid 4 to smithi10428818-23 2020-03-09T00:25:18.951 INFO:tasks.rados.rados.0.smithi104.stdout:221: finishing write tid 6 to smithi10428818-23 2020-03-09T00:25:18.951 INFO:tasks.rados.rados.0.smithi104.stdout:223: finishing copy_from to smithi10428818-35 2020-03-09T00:25:18.951 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 35 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-03-09T00:25:18.951 INFO:tasks.rados.rados.0.smithi104.stdout:221: finishing write tid 7 to smithi10428818-23 2020-03-09T00:25:18.951 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 23 v 52 (ObjNum 101 snap 17 seq_num 101) dirty exists 2020-03-09T00:25:18.951 INFO:tasks.rados.rados.0.smithi104.stdout:221: left oid 23 (ObjNum 101 snap 17 seq_num 101) 2020-03-09T00:25:18.952 INFO:tasks.rados.rados.0.smithi104.stdout:223: finishing copy_from racing read to smithi10428818-35 2020-03-09T00:25:18.952 INFO:tasks.rados.rados.0.smithi104.stdout:225: finishing write tid 1 to smithi10428818-11 2020-03-09T00:25:18.952 INFO:tasks.rados.rados.0.smithi104.stdout:225: finishing write tid 2 to smithi10428818-11 2020-03-09T00:25:18.952 INFO:tasks.rados.rados.0.smithi104.stdout:225: finishing write tid 3 to smithi10428818-11 2020-03-09T00:25:18.952 INFO:tasks.rados.rados.0.smithi104.stdout:225: finishing write tid 5 to smithi10428818-11 2020-03-09T00:25:18.952 INFO:tasks.rados.rados.0.smithi104.stdout:226: finishing copy_from to smithi10428818-46 2020-03-09T00:25:18.952 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 46 v 19 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-03-09T00:25:18.953 INFO:tasks.rados.rados.0.smithi104.stdout:225: finishing write tid 6 to smithi10428818-11 2020-03-09T00:25:18.953 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 11 v 69 (ObjNum 102 snap 17 seq_num 102) dirty exists 2020-03-09T00:25:18.953 INFO:tasks.rados.rados.0.smithi104.stdout:225: left oid 11 (ObjNum 102 snap 17 seq_num 102) 2020-03-09T00:25:18.953 INFO:tasks.rados.rados.0.smithi104.stdout:226: finishing copy_from racing read to smithi10428818-46 2020-03-09T00:25:18.953 INFO:tasks.rados.rados.0.smithi104.stdout:227: finishing write tid 1 to smithi10428818-22 2020-03-09T00:25:18.953 INFO:tasks.rados.rados.0.smithi104.stdout:227: finishing write tid 2 to smithi10428818-22 2020-03-09T00:25:18.953 INFO:tasks.rados.rados.0.smithi104.stdout:227: finishing write tid 3 to smithi10428818-22 2020-03-09T00:25:18.954 INFO:tasks.rados.rados.0.smithi104.stdout:227: finishing write tid 5 to smithi10428818-22 2020-03-09T00:25:18.954 INFO:tasks.rados.rados.0.smithi104.stdout:227: finishing write tid 6 to smithi10428818-22 2020-03-09T00:25:18.954 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 22 v 131 (ObjNum 103 snap 17 seq_num 103) dirty exists 2020-03-09T00:25:18.954 INFO:tasks.rados.rados.0.smithi104.stdout:227: left oid 22 (ObjNum 103 snap 17 seq_num 103) 2020-03-09T00:25:18.954 INFO:tasks.rados.rados.0.smithi104.stdout:229: finishing write tid 1 to smithi10428818-40 2020-03-09T00:25:18.954 INFO:tasks.rados.rados.0.smithi104.stdout:229: finishing write tid 2 to smithi10428818-40 2020-03-09T00:25:18.954 INFO:tasks.rados.rados.0.smithi104.stdout:229: finishing write tid 3 to smithi10428818-40 2020-03-09T00:25:18.955 INFO:tasks.rados.rados.0.smithi104.stdout:229: finishing write tid 5 to smithi10428818-40 2020-03-09T00:25:18.955 INFO:tasks.rados.rados.0.smithi104.stdout:230: finishing write tid 1 to smithi10428818-21 2020-03-09T00:25:18.955 INFO:tasks.rados.rados.0.smithi104.stdout:221: done (8 left) 2020-03-09T00:25:18.955 INFO:tasks.rados.rados.0.smithi104.stdout:223: done (7 left) 2020-03-09T00:25:18.955 INFO:tasks.rados.rados.0.smithi104.stdout:224: done (6 left) 2020-03-09T00:25:18.955 INFO:tasks.rados.rados.0.smithi104.stdout:225: done (5 left) 2020-03-09T00:25:18.955 INFO:tasks.rados.rados.0.smithi104.stdout:226: done (4 left) 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:227: done (3 left) 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:233: delete oid 42 current snap is 17 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:230: finishing write tid 2 to smithi10428818-21 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:229: finishing write tid 6 to smithi10428818-40 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 40 v 46 (ObjNum 104 snap 17 seq_num 104) dirty exists 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:229: left oid 40 (ObjNum 104 snap 17 seq_num 104) 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:230: finishing write tid 3 to smithi10428818-21 2020-03-09T00:25:18.956 INFO:tasks.rados.rados.0.smithi104.stdout:230: finishing write tid 5 to smithi10428818-21 2020-03-09T00:25:18.957 INFO:tasks.rados.rados.0.smithi104.stdout:230: finishing write tid 6 to smithi10428818-21 2020-03-09T00:25:18.957 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 21 v 26 (ObjNum 105 snap 17 seq_num 105) dirty exists 2020-03-09T00:25:18.957 INFO:tasks.rados.rados.0.smithi104.stdout:230: left oid 21 (ObjNum 105 snap 17 seq_num 105) 2020-03-09T00:25:18.957 INFO:tasks.rados.rados.0.smithi104.stdout:229: done (3 left) 2020-03-09T00:25:18.957 INFO:tasks.rados.rados.0.smithi104.stdout:230: done (2 left) 2020-03-09T00:25:18.957 INFO:tasks.rados.rados.0.smithi104.stdout:233: done (1 left) 2020-03-09T00:25:18.957 INFO:tasks.rados.rados.0.smithi104.stdout:234: read oid 10 snap -1 2020-03-09T00:25:18.958 INFO:tasks.rados.rados.0.smithi104.stdout:234: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:25:18.958 INFO:tasks.rados.rados.0.smithi104.stdout:235: read oid 4 snap -1 2020-03-09T00:25:18.958 INFO:tasks.rados.rados.0.smithi104.stdout:235: expect (ObjNum 34 snap 0 seq_num 34) 2020-03-09T00:25:18.958 INFO:tasks.rados.rados.0.smithi104.stdout:236: read oid 1 snap -1 2020-03-09T00:25:18.958 INFO:tasks.rados.rados.0.smithi104.stdout:236: expect (ObjNum 0 snap 0 seq_num 0) 2020-03-09T00:25:18.958 INFO:tasks.rados.rados.0.smithi104.stdout:237: rollback oid 37 current snap is 17 2020-03-09T00:25:18.958 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 37 to 16 2020-03-09T00:25:18.959 INFO:tasks.rados.rados.0.smithi104.stdout:238: snap_create 2020-03-09T00:25:18.959 INFO:tasks.rados.rados.0.smithi104.stdout:232: finishing rollback tid 0 to smithi10428818-28 2020-03-09T00:25:18.959 INFO:tasks.rados.rados.0.smithi104.stdout:232: finishing rollback tid 1 to smithi10428818-28 2020-03-09T00:25:18.959 INFO:tasks.rados.rados.0.smithi104.stdout:232: finishing rollback tid 2 to smithi10428818-28 2020-03-09T00:25:18.959 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 28 v 57 (ObjNum 89 snap 14 seq_num 89) dirty exists 2020-03-09T00:25:18.959 INFO:tasks.rados.rados.0.smithi104.stdout:237: finishing rollback tid 0 to smithi10428818-37 2020-03-09T00:25:18.959 INFO:tasks.rados.rados.0.smithi104.stdout:237: finishing rollback tid 1 to smithi10428818-37 2020-03-09T00:25:18.960 INFO:tasks.rados.rados.0.smithi104.stdout:237: finishing rollback tid 2 to smithi10428818-37 2020-03-09T00:25:18.960 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 37 v 135 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-03-09T00:25:18.960 INFO:tasks.rados.rados.0.smithi104.stdout:235: expect (ObjNum 34 snap 0 seq_num 34) 2020-03-09T00:25:18.960 INFO:tasks.rados.rados.0.smithi104.stdout:236: expect (ObjNum 0 snap 0 seq_num 0) 2020-03-09T00:25:18.961 INFO:tasks.rados.rados.0.smithi104.stdout:234: expect (ObjNum 52 snap 0 seq_num 52) 2020-03-09T00:25:18.961 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is set 2020-03-09T00:25:18.961 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:18.804+0000 7efc11cac700 -1 osd.8 pg_epoch: 69 pg[3.es0( v 69'96 (0'0,69'96] local-lis/les=32/33 n=7 ec=32/32 lis/c=32/32 les/c/f=33/33/0 sis=32) [8,9,1,11,2,0,3]p8(0) r=0 lpr=32 crt=69'96 lcod 69'94 mlcod 69'94 active+clean+snaptrim trimq=[13~1] ps=[5~1,c~1,15~1,1b~1]] removing snap head 2020-03-09T00:25:18.961 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:18.942+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:18.962 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:25:19.045 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:19.044+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:19.077 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:19.078 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:19.078 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:19.086 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:25:19.146 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:19.145+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:19.205 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:19.205 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:19.205 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:19.205 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:19.205 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:19.211 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:25:19.247 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:19.245+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:19.348 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:19.347+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:19.366 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:19.367 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:19.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:19.367 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:19.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.367 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:196 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~540672] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:19.367 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:19.367 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.5385448, 2020-03-09T00:25:19.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501, 2020-03-09T00:25:19.368 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.368 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:19.368 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.368 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.368 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.368 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 196 2020-03-09T00:25:19.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.369 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:19.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:19.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035415000000000002 2020-03-09T00:25:19.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826046+0000", 2020-03-09T00:25:19.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2260000000000005e-06 2020-03-09T00:25:19.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826054+0000", 2020-03-09T00:25:19.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3873000000000001e-05 2020-03-09T00:25:19.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826068+0000", 2020-03-09T00:25:19.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2020-03-09T00:25:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826074+0000", 2020-03-09T00:25:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0126e-05 2020-03-09T00:25:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826134+0000", 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001117 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826246+0000", 2020-03-09T00:25:19.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00124165 2020-03-09T00:25:19.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827488+0000", 2020-03-09T00:25:19.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088469059999999999 2020-03-09T00:25:19.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836335+0000", 2020-03-09T00:25:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035733999999999999 2020-03-09T00:25:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836692+0000", 2020-03-09T00:25:19.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1440000000000005e-05 2020-03-09T00:25:19.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836783+0000", 2020-03-09T00:25:19.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501 2020-03-09T00:25:19.377 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.378 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.378 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:197 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:19.379 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:19.379 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.533286263000001, 2020-03-09T00:25:19.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097135529999999998, 2020-03-09T00:25:19.379 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.379 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:19.379 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.380 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.380 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 197 2020-03-09T00:25:19.380 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.380 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.381 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:19.381 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.381 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.381 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.381 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:19.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027437700000000002 2020-03-09T00:25:19.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.382 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.382 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831225+0000", 2020-03-09T00:25:19.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8239999999999996e-06 2020-03-09T00:25:19.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.383 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.383 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.383 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831231+0000", 2020-03-09T00:25:19.383 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3345e-05 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831244+0000", 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4499999999999998e-06 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831251+0000", 2020-03-09T00:25:19.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057306900000000005 2020-03-09T00:25:19.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831824+0000", 2020-03-09T00:25:19.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103781 2020-03-09T00:25:19.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831928+0000", 2020-03-09T00:25:19.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000811518 2020-03-09T00:25:19.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832740+0000", 2020-03-09T00:25:19.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036368210000000002 2020-03-09T00:25:19.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836377+0000", 2020-03-09T00:25:19.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042093749999999996 2020-03-09T00:25:19.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:19.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840586+0000", 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7992999999999997e-05 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840664+0000", 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097135529999999998 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.389 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.390 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.390 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:198 3.5s0 3:ada90060:::smithi10428818-8:head [setxattr _header (55) in=7b] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:19.391 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:19.391 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.532930578999999, 2020-03-09T00:25:19.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095424670000000007, 2020-03-09T00:25:19.391 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.391 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:19.391 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.391 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.392 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.392 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198 2020-03-09T00:25:19.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.392 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.392 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.392 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:19.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.393 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.393 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.393 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:19.393 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9859999999999999e-05 2020-03-09T00:25:19.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831376+0000", 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3319999999999997e-06 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831382+0000", 2020-03-09T00:25:19.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1367e-05 2020-03-09T00:25:19.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831394+0000", 2020-03-09T00:25:19.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4580000000000003e-06 2020-03-09T00:25:19.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831398+0000", 2020-03-09T00:25:19.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039429479999999999 2020-03-09T00:25:19.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835341+0000", 2020-03-09T00:25:19.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011724 2020-03-09T00:25:19.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835458+0000", 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068325099999999995 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836141+0000", 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092122099999999995 2020-03-09T00:25:19.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.837063+0000", 2020-03-09T00:25:19.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036999060000000002 2020-03-09T00:25:19.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:19.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840763+0000", 2020-03-09T00:25:19.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5884000000000006e-05 2020-03-09T00:25:19.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840848+0000", 2020-03-09T00:25:19.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095424670000000007 2020-03-09T00:25:19.403 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.403 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.403 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.403 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:199 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:19.404 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:19.404 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.532798680999999, 2020-03-09T00:25:19.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149, 2020-03-09T00:25:19.404 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.404 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:19.404 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.404 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 199 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:19.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:19.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1242999999999998e-05 2020-03-09T00:25:19.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831479+0000", 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5740000000000001e-06 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831486+0000", 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1513e-05 2020-03-09T00:25:19.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.409 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831497+0000", 2020-03-09T00:25:19.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2760000000000002e-06 2020-03-09T00:25:19.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.409 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831502+0000", 2020-03-09T00:25:19.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053448499999999999 2020-03-09T00:25:19.410 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.410 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.410 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.410 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836846+0000", 2020-03-09T00:25:19.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103844 2020-03-09T00:25:19.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.411 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:19.411 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836950+0000", 2020-03-09T00:25:19.411 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045940299999999998 2020-03-09T00:25:19.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.412 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.413 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841544+0000", 2020-03-09T00:25:19.413 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105018 2020-03-09T00:25:19.413 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.413 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841649+0000", 2020-03-09T00:25:19.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002376801 2020-03-09T00:25:19.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.844026+0000", 2020-03-09T00:25:19.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:346 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:19.415 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.538705084, 2020-03-09T00:25:19.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654, 2020-03-09T00:25:19.416 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.416 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:19.416 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.416 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.416 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.416 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 346 2020-03-09T00:25:19.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.417 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.417 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:19.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014902900000000001 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825681+0000", 2020-03-09T00:25:19.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:19.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825687+0000", 2020-03-09T00:25:19.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6099999999999995e-06 2020-03-09T00:25:19.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825696+0000", 2020-03-09T00:25:19.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6149999999999999e-06 2020-03-09T00:25:19.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825701+0000", 2020-03-09T00:25:19.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4975000000000002e-05 2020-03-09T00:25:19.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825746+0000", 2020-03-09T00:25:19.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0597e-05 2020-03-09T00:25:19.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825836+0000", 2020-03-09T00:25:19.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091597100000000002 2020-03-09T00:25:19.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.423 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.423 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826752+0000", 2020-03-09T00:25:19.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046291539999999999 2020-03-09T00:25:19.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.424 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831382+0000", 2020-03-09T00:25:19.424 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098416760000000006 2020-03-09T00:25:19.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.424 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.424 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:19.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841223+0000", 2020-03-09T00:25:19.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100926 2020-03-09T00:25:19.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841324+0000", 2020-03-09T00:25:19.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:347 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~622592] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:19.426 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.531437262000001, 2020-03-09T00:25:19.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999, 2020-03-09T00:25:19.427 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.427 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:19.427 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.427 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.427 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.427 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 347 2020-03-09T00:25:19.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.428 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:19.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:19.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000469071 2020-03-09T00:25:19.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833268+0000", 2020-03-09T00:25:19.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7889999999999995e-06 2020-03-09T00:25:19.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833276+0000", 2020-03-09T00:25:19.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6033999999999999e-05 2020-03-09T00:25:19.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.432 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833292+0000", 2020-03-09T00:25:19.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8850000000000001e-06 2020-03-09T00:25:19.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.433 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.433 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833299+0000", 2020-03-09T00:25:19.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:19.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833343+0000", 2020-03-09T00:25:19.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105574 2020-03-09T00:25:19.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833448+0000", 2020-03-09T00:25:19.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094406100000000005 2020-03-09T00:25:19.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834392+0000", 2020-03-09T00:25:19.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088904150000000005 2020-03-09T00:25:19.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843283+0000", 2020-03-09T00:25:19.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052783500000000004 2020-03-09T00:25:19.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:19.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843811+0000", 2020-03-09T00:25:19.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1547999999999998e-05 2020-03-09T00:25:19.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843902+0000", 2020-03-09T00:25:19.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999 2020-03-09T00:25:19.438 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.438 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:348 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~147456] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.530881172000001, 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999, 2020-03-09T00:25:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.440 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:19.440 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.440 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.440 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.440 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 348 2020-03-09T00:25:19.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.440 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:19.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019096900000000001 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833546+0000", 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4060000000000002e-06 2020-03-09T00:25:19.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.443 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833553+0000", 2020-03-09T00:25:19.443 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1321000000000001e-05 2020-03-09T00:25:19.443 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.445 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:19.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6310000000000002e-06 2020-03-09T00:25:19.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833569+0000", 2020-03-09T00:25:19.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037440260000000001 2020-03-09T00:25:19.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837313+0000", 2020-03-09T00:25:19.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134988 2020-03-09T00:25:19.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837448+0000", 2020-03-09T00:25:19.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001155454 2020-03-09T00:25:19.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.838603+0000", 2020-03-09T00:25:19.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046829050000000002 2020-03-09T00:25:19.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843286+0000", 2020-03-09T00:25:19.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011927330000000001 2020-03-09T00:25:19.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:19.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844479+0000", 2020-03-09T00:25:19.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5680000000000005e-05 2020-03-09T00:25:19.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844575+0000", 2020-03-09T00:25:19.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999 2020-03-09T00:25:19.453 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.453 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.453 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.454 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:349 3.5s0 3:ada90060:::smithi10428818-8:head [setxattr _header (56) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:19.454 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:19.454 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.530615611999998, 2020-03-09T00:25:19.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001, 2020-03-09T00:25:19.454 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.455 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:19.455 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.455 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.455 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.456 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 349 2020-03-09T00:25:19.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.456 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.457 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.457 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:19.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.457 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.457 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:19.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8177999999999997e-05 2020-03-09T00:25:19.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833689+0000", 2020-03-09T00:25:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3619999999999997e-06 2020-03-09T00:25:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833695+0000", 2020-03-09T00:25:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1154e-05 2020-03-09T00:25:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833706+0000", 2020-03-09T00:25:19.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5030000000000002e-06 2020-03-09T00:25:19.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833712+0000", 2020-03-09T00:25:19.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0079814919999999998 2020-03-09T00:25:19.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841694+0000", 2020-03-09T00:25:19.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013282600000000001 2020-03-09T00:25:19.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841826+0000", 2020-03-09T00:25:19.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071321600000000002 2020-03-09T00:25:19.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.842540+0000", 2020-03-09T00:25:19.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074733700000000004 2020-03-09T00:25:19.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843287+0000", 2020-03-09T00:25:19.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001845368 2020-03-09T00:25:19.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:19.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845132+0000", 2020-03-09T00:25:19.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000004e-05 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845219+0000", 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.465 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.466 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:350 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:19.466 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:19.466 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 16.530485955, 2020-03-09T00:25:19.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417, 2020-03-09T00:25:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 350 2020-03-09T00:25:19.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.468 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.468 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.468 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:19.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:19.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0074999999999998e-05 2020-03-09T00:25:19.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833791+0000", 2020-03-09T00:25:19.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1530000000000003e-06 2020-03-09T00:25:19.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833800+0000", 2020-03-09T00:25:19.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3621000000000001e-05 2020-03-09T00:25:19.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833813+0000", 2020-03-09T00:25:19.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9899999999999999e-06 2020-03-09T00:25:19.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833817+0000", 2020-03-09T00:25:19.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0093306459999999997 2020-03-09T00:25:19.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843148+0000", 2020-03-09T00:25:19.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012790100000000001 2020-03-09T00:25:19.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:19.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843276+0000", 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025486789999999999 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845825+0000", 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120695 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845945+0000", 2020-03-09T00:25:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033866569999999999 2020-03-09T00:25:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.849332+0000", 2020-03-09T00:25:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417 2020-03-09T00:25:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s4 54/48 ECSubWrite(tid=8, reqid=client.4751.0:376, at_version=54'17, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.809652+0000", 2020-03-09T00:25:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.554584103, 2020-03-09T00:25:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083668240000000001, 2020-03-09T00:25:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.477 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809652+0000", 2020-03-09T00:25:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809652+0000", 2020-03-09T00:25:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031753700000000001 2020-03-09T00:25:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809970+0000", 2020-03-09T00:25:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2459999999999998e-05 2020-03-09T00:25:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809992+0000", 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8159999999999999e-05 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810011+0000", 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0770000000000008e-06 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810019+0000", 2020-03-09T00:25:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6953999999999997e-05 2020-03-09T00:25:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810076+0000", 2020-03-09T00:25:19.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2466999999999999e-05 2020-03-09T00:25:19.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810088+0000", 2020-03-09T00:25:19.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0078815440000000007 2020-03-09T00:25:19.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.817970+0000", 2020-03-09T00:25:19.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9625e-05 2020-03-09T00:25:19.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818019+0000", 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083668240000000001 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.485 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s3 59/54 ECSubWrite(tid=26, reqid=osd.0.0:26, at_version=59'36, trim_to=0'0, roll_forward_to=58'32))", 2020-03-09T00:25:19.486 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.815646+0000", 2020-03-09T00:25:19.486 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.548591024, 2020-03-09T00:25:19.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088469659999999995, 2020-03-09T00:25:19.486 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.486 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.486 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815646+0000", 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815646+0000", 2020-03-09T00:25:19.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7358999999999997e-05 2020-03-09T00:25:19.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815703+0000", 2020-03-09T00:25:19.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6523e-05 2020-03-09T00:25:19.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815729+0000", 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3370000000000001e-05 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815743+0000", 2020-03-09T00:25:19.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4940000000000003e-06 2020-03-09T00:25:19.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815748+0000", 2020-03-09T00:25:19.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001217587 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.816966+0000", 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2965999999999999e-05 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.816979+0000", 2020-03-09T00:25:19.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074674629999999997 2020-03-09T00:25:19.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.824446+0000", 2020-03-09T00:25:19.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6204e-05 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.824493+0000", 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088469659999999995 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.493 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.494 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.8s5 59/54 ECSubWrite(tid=5, reqid=client.4751.0:485, at_version=59'26, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:19.494 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.815672+0000", 2020-03-09T00:25:19.494 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.548564963, 2020-03-09T00:25:19.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088535950000000006, 2020-03-09T00:25:19.494 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.495 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.495 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815672+0000", 2020-03-09T00:25:19.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815672+0000", 2020-03-09T00:25:19.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6192e-05 2020-03-09T00:25:19.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815728+0000", 2020-03-09T00:25:19.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8292999999999999e-05 2020-03-09T00:25:19.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815746+0000", 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3382e-05 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815759+0000", 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1229999999999997e-06 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815766+0000", 2020-03-09T00:25:19.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8077999999999993e-05 2020-03-09T00:25:19.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815844+0000", 2020-03-09T00:25:19.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3181999999999999e-05 2020-03-09T00:25:19.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.815867+0000", 2020-03-09T00:25:19.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0086334289999999998 2020-03-09T00:25:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.824500+0000", 2020-03-09T00:25:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4916e-05 2020-03-09T00:25:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.824525+0000", 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088535950000000006 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.503 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.8s5 59/54 ECSubWrite(tid=6, reqid=client.4751.0:486, at_version=59'27, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:19.503 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:19.503 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.545345153, 2020-03-09T00:25:19.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012816759, 2020-03-09T00:25:19.503 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.503 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.503 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:19.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102045 2020-03-09T00:25:19.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818993+0000", 2020-03-09T00:25:19.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2311999999999999e-05 2020-03-09T00:25:19.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819016+0000", 2020-03-09T00:25:19.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6019999999999999e-05 2020-03-09T00:25:19.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819032+0000", 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4910000000000002e-06 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819038+0000", 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023069140000000002 2020-03-09T00:25:19.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821345+0000", 2020-03-09T00:25:19.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3864e-05 2020-03-09T00:25:19.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821359+0000", 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010305182 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831664+0000", 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3930999999999997e-05 2020-03-09T00:25:19.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.510 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.510 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831708+0000", 2020-03-09T00:25:19.510 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012816759 2020-03-09T00:25:19.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.510 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.511 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 59/54 ECSubWrite(tid=159, reqid=osd.12.0:159, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:19.511 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.821409+0000", 2020-03-09T00:25:19.511 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.5428278, 2020-03-09T00:25:19.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010250832, 2020-03-09T00:25:19.512 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.512 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.512 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821409+0000", 2020-03-09T00:25:19.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821409+0000", 2020-03-09T00:25:19.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8247e-05 2020-03-09T00:25:19.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821467+0000", 2020-03-09T00:25:19.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4675e-05 2020-03-09T00:25:19.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821492+0000", 2020-03-09T00:25:19.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5407e-05 2020-03-09T00:25:19.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821507+0000", 2020-03-09T00:25:19.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1920000000000002e-06 2020-03-09T00:25:19.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821513+0000", 2020-03-09T00:25:19.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102107 2020-03-09T00:25:19.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821615+0000", 2020-03-09T00:25:19.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3492999999999999e-05 2020-03-09T00:25:19.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821629+0000", 2020-03-09T00:25:19.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0099815000000000008 2020-03-09T00:25:19.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831610+0000", 2020-03-09T00:25:19.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9211e-05 2020-03-09T00:25:19.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831660+0000", 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010250832 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.521 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s4 59/48 ECSubWrite(tid=23, reqid=osd.9.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=59'30))", 2020-03-09T00:25:19.521 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.822203+0000", 2020-03-09T00:25:19.521 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.542033392, 2020-03-09T00:25:19.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.009406078, 2020-03-09T00:25:19.521 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.521 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.522 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822203+0000", 2020-03-09T00:25:19.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822203+0000", 2020-03-09T00:25:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8763e-05 2020-03-09T00:25:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822262+0000", 2020-03-09T00:25:19.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4282e-05 2020-03-09T00:25:19.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822276+0000", 2020-03-09T00:25:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5724000000000001e-05 2020-03-09T00:25:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822292+0000", 2020-03-09T00:25:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2359999999999998e-06 2020-03-09T00:25:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822299+0000", 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000229817 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822529+0000", 2020-03-09T00:25:19.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3582e-05 2020-03-09T00:25:19.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822543+0000", 2020-03-09T00:25:19.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.009010611 2020-03-09T00:25:19.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831553+0000", 2020-03-09T00:25:19.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6063000000000002e-05 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831609+0000", 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.009406078 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.529 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 59/32 ECSubWrite(tid=160, reqid=osd.12.0:160, at_version=59'77, trim_to=0'0, roll_forward_to=58'75))", 2020-03-09T00:25:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.825104+0000", 2020-03-09T00:25:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.539132512, 2020-03-09T00:25:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092538810000000003, 2020-03-09T00:25:19.530 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825104+0000", 2020-03-09T00:25:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825104+0000", 2020-03-09T00:25:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2480999999999999e-05 2020-03-09T00:25:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825187+0000", 2020-03-09T00:25:19.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2126000000000001e-05 2020-03-09T00:25:19.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825209+0000", 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5880999999999999e-05 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825225+0000", 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4710000000000004e-06 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825231+0000", 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020443600000000001 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825435+0000", 2020-03-09T00:25:19.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.258e-05 2020-03-09T00:25:19.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825448+0000", 2020-03-09T00:25:19.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088661420000000005 2020-03-09T00:25:19.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.834314+0000", 2020-03-09T00:25:19.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3764000000000003e-05 2020-03-09T00:25:19.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.834358+0000", 2020-03-09T00:25:19.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092538810000000003 2020-03-09T00:25:19.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.539 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.539 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 59/54 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=59'43, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:19.540 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.825378+0000", 2020-03-09T00:25:19.540 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.538858477, 2020-03-09T00:25:19.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0089450120000000004, 2020-03-09T00:25:19.540 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.540 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.540 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825378+0000", 2020-03-09T00:25:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825378+0000", 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2692e-05 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825461+0000", 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8717000000000001e-05 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825480+0000", 2020-03-09T00:25:19.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.434e-05 2020-03-09T00:25:19.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825494+0000", 2020-03-09T00:25:19.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.344e-06 2020-03-09T00:25:19.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825499+0000", 2020-03-09T00:25:19.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6826e-05 2020-03-09T00:25:19.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825566+0000", 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3463e-05 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825579+0000", 2020-03-09T00:25:19.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0086899030000000006 2020-03-09T00:25:19.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.834269+0000", 2020-03-09T00:25:19.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3727000000000002e-05 2020-03-09T00:25:19.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.834323+0000", 2020-03-09T00:25:19.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0089450120000000004 2020-03-09T00:25:19.547 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.547 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.547 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 59/32 ECSubWrite(tid=162, reqid=osd.12.0:162, at_version=59'79, trim_to=0'0, roll_forward_to=58'75))", 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.538639195, 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013484734, 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.548 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.549 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:19.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:19.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120926 2020-03-09T00:25:19.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825718+0000", 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8563000000000003e-05 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825757+0000", 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3974999999999999e-05 2020-03-09T00:25:19.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825791+0000", 2020-03-09T00:25:19.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2510999999999999e-05 2020-03-09T00:25:19.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825803+0000", 2020-03-09T00:25:19.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062712599999999999 2020-03-09T00:25:19.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826430+0000", 2020-03-09T00:25:19.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2381000000000001e-05 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826443+0000", 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012584521 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839027+0000", 2020-03-09T00:25:19.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4731000000000002e-05 2020-03-09T00:25:19.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839082+0000", 2020-03-09T00:25:19.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013484734 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 59/32 ECSubWrite(tid=163, reqid=client.4751.0:488, at_version=59'80, trim_to=0'0, roll_forward_to=58'75))", 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.829002+0000", 2020-03-09T00:25:19.556 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.535234682, 2020-03-09T00:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010120809, 2020-03-09T00:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829002+0000", 2020-03-09T00:25:19.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829002+0000", 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018660200000000001 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829189+0000", 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3999999999999999e-05 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829233+0000", 2020-03-09T00:25:19.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2617000000000002e-05 2020-03-09T00:25:19.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829275+0000", 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1619999999999996e-06 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829281+0000", 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002744841 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.832026+0000", 2020-03-09T00:25:19.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2108e-05 2020-03-09T00:25:19.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.832038+0000", 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0070503889999999998 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839089+0000", 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4090000000000001e-05 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839123+0000", 2020-03-09T00:25:19.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010120809 2020-03-09T00:25:19.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.564 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.565 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as5 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:19.565 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.914442+0000", 2020-03-09T00:25:19.565 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.449794881000001, 2020-03-09T00:25:19.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0093004049999999994, 2020-03-09T00:25:19.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:19.565 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:19.566 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:19.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:19.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914442+0000", 2020-03-09T00:25:19.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:19.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:19.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914442+0000", 2020-03-09T00:25:19.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000173905 2020-03-09T00:25:19.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:19.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914616+0000", 2020-03-09T00:25:19.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9518000000000001e-05 2020-03-09T00:25:19.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:19.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914645+0000", 2020-03-09T00:25:19.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6934000000000001e-05 2020-03-09T00:25:19.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:19.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914662+0000", 2020-03-09T00:25:19.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5327000000000001e-05 2020-03-09T00:25:19.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914677+0000", 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1156999999999997e-05 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914739+0000", 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3625000000000001e-05 2020-03-09T00:25:19.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:19.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914752+0000", 2020-03-09T00:25:19.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0089546320000000006 2020-03-09T00:25:19.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:19.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923707+0000", 2020-03-09T00:25:19.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5306999999999999e-05 2020-03-09T00:25:19.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:19.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:19.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:19.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923742+0000", 2020-03-09T00:25:19.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0093004049999999994 2020-03-09T00:25:19.575 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.575 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.575 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.575 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:19.575 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:19.575 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:19.577 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:19.575+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:19.578 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 2.7 2020-03-09T00:25:19.579 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:25:19.677 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:19.676+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:19.761 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:19.761 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:19.761 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:19.761 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:19.770 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:25:19.779 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:19.776+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:19.796 INFO:tasks.rados.rados.0.smithi104.stdout:232: done (5 left) 2020-03-09T00:25:19.796 INFO:tasks.rados.rados.0.smithi104.stdout:234: done (4 left) 2020-03-09T00:25:19.796 INFO:tasks.rados.rados.0.smithi104.stdout:235: done (3 left) 2020-03-09T00:25:19.797 INFO:tasks.rados.rados.0.smithi104.stdout:236: done (2 left) 2020-03-09T00:25:19.797 INFO:tasks.rados.rados.0.smithi104.stdout:237: done (1 left) 2020-03-09T00:25:19.797 INFO:tasks.rados.rados.0.smithi104.stdout:238: done (0 left) 2020-03-09T00:25:19.797 INFO:tasks.rados.rados.0.smithi104.stdout:239: snap_create 2020-03-09T00:25:19.884 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:19.878+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:19.887 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:19.888 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:19.888 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:19.888 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:19.888 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:19.896 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:25:19.913 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:19.981 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:19.980+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:20.014 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:20.014 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:20.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:20.015 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:20.015 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.015 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:311 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~770048] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:20.015 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:20.015 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.227604125999999, 2020-03-09T00:25:20.015 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438, 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 311 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.016 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.020 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.021 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:20.021 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.021 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.021 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.021 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014001199999999999 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785132+0000", 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1973999999999999e-05 2020-03-09T00:25:20.022 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.023 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785144+0000", 2020-03-09T00:25:20.023 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1815e-05 2020-03-09T00:25:20.023 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.023 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.023 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785156+0000", 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2449999999999998e-06 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785161+0000", 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00063616300000000003 2020-03-09T00:25:20.024 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.025 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785797+0000", 2020-03-09T00:25:20.025 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00065134999999999998 2020-03-09T00:25:20.025 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.025 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.025 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.786449+0000", 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00188737 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788336+0000", 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010887625999999999 2020-03-09T00:25:20.026 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.027 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.027 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.027 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799224+0000", 2020-03-09T00:25:20.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037060560000000001 2020-03-09T00:25:20.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.032 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.032 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.032 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.802930+0000", 2020-03-09T00:25:20.032 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000460719 2020-03-09T00:25:20.032 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.033 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.033 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.033 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803390+0000", 2020-03-09T00:25:20.033 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438 2020-03-09T00:25:20.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.033 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.033 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:312 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~376832] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.226107978000002, 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001, 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.034 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.035 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.035 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 312 2020-03-09T00:25:20.035 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.035 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.035 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:20.036 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026595500000000001 2020-03-09T00:25:20.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.037 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.037 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785494+0000", 2020-03-09T00:25:20.037 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3902e-05 2020-03-09T00:25:20.037 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.037 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785508+0000", 2020-03-09T00:25:20.038 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.417e-05 2020-03-09T00:25:20.038 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.038 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.038 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.038 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785522+0000", 2020-03-09T00:25:20.039 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.99e-06 2020-03-09T00:25:20.039 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785529+0000", 2020-03-09T00:25:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083854199999999993 2020-03-09T00:25:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793915+0000", 2020-03-09T00:25:20.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024809000000000002 2020-03-09T00:25:20.041 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.041 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.041 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.041 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.794163+0000", 2020-03-09T00:25:20.041 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001122184 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795285+0000", 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004034358 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.042 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799319+0000", 2020-03-09T00:25:20.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042626369999999997 2020-03-09T00:25:20.043 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.043 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.043 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.043 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803582+0000", 2020-03-09T00:25:20.043 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00042245199999999997 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804004+0000", 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.044 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.045 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.045 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.045 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:313 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:20.045 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:20.045 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.225752416999999, 2020-03-09T00:25:20.045 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001, 2020-03-09T00:25:20.045 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 313 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.046 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:20.047 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.047 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.047 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.047 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.047 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7169000000000004e-05 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785681+0000", 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8849999999999999e-06 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.048 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.049 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.049 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785689+0000", 2020-03-09T00:25:20.049 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5290000000000001e-05 2020-03-09T00:25:20.049 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.049 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.050 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.050 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785704+0000", 2020-03-09T00:25:20.050 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5759999999999996e-06 2020-03-09T00:25:20.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.051 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.051 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785710+0000", 2020-03-09T00:25:20.051 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012205476 2020-03-09T00:25:20.051 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.051 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.797915+0000", 2020-03-09T00:25:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023467400000000001 2020-03-09T00:25:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.052 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.798150+0000", 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00085795900000000004 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799008+0000", 2020-03-09T00:25:20.053 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00050973499999999998 2020-03-09T00:25:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799518+0000", 2020-03-09T00:25:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046306960000000001 2020-03-09T00:25:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.054 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804148+0000", 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00032100500000000002 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804469+0000", 2020-03-09T00:25:20.055 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001 2020-03-09T00:25:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:314 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:20.056 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.225569637, 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998, 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.057 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 314 2020-03-09T00:25:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.058 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9129999999999999e-05 2020-03-09T00:25:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.059 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.060 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785816+0000", 2020-03-09T00:25:20.061 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1280000000000002e-06 2020-03-09T00:25:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.062 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785823+0000", 2020-03-09T00:25:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.24e-05 2020-03-09T00:25:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.063 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785835+0000", 2020-03-09T00:25:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1800000000000004e-06 2020-03-09T00:25:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.064 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785840+0000", 2020-03-09T00:25:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014080127 2020-03-09T00:25:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799920+0000", 2020-03-09T00:25:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023000599999999999 2020-03-09T00:25:20.065 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.066 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.800150+0000", 2020-03-09T00:25:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048975440000000002 2020-03-09T00:25:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.067 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805048+0000", 2020-03-09T00:25:20.068 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000221747 2020-03-09T00:25:20.068 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.068 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.068 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.068 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805270+0000", 2020-03-09T00:25:20.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037657239999999998 2020-03-09T00:25:20.069 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.069 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.069 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.069 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.809036+0000", 2020-03-09T00:25:20.069 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998 2020-03-09T00:25:20.069 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.070 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.070 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.070 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.070 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.070 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:20.075 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:20.075 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.159814598000001, 2020-03-09T00:25:20.076 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001, 2020-03-09T00:25:20.076 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.076 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:20.076 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.076 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.076 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.076 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:20.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.077 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.077 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.077 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:20.077 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0030360970000000002 2020-03-09T00:25:20.077 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.078 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854558+0000", 2020-03-09T00:25:20.078 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8904e-05 2020-03-09T00:25:20.078 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.078 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.078 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854587+0000", 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5423000000000001e-05 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854602+0000", 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9139999999999997e-06 2020-03-09T00:25:20.079 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854611+0000", 2020-03-09T00:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2179000000000002e-05 2020-03-09T00:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.080 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854673+0000", 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6330000000000001e-05 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854689+0000", 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019248372999999999 2020-03-09T00:25:20.081 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873938+0000", 2020-03-09T00:25:20.082 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3464000000000003e-05 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873981+0000", 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.083 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:397 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~573440] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.183027286000002, 2020-03-09T00:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99949370299999996, 2020-03-09T00:25:20.084 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 397 2020-03-09T00:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:20.086 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015862319999999999 2020-03-09T00:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.087 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829895+0000", 2020-03-09T00:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1249999999999999e-06 2020-03-09T00:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.088 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829904+0000", 2020-03-09T00:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5704000000000001e-05 2020-03-09T00:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829920+0000", 2020-03-09T00:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3140000000000004e-06 2020-03-09T00:25:20.089 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829928+0000", 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000122717 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.090 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830051+0000", 2020-03-09T00:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9819999999999999e-06 2020-03-09T00:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830056+0000", 2020-03-09T00:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97235390200000005 2020-03-09T00:25:20.091 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.802410+0000", 2020-03-09T00:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016568099999999999 2020-03-09T00:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.802576+0000", 2020-03-09T00:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001235402 2020-03-09T00:25:20.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.803811+0000", 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013426433999999999 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817237+0000", 2020-03-09T00:25:20.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010489160000000001 2020-03-09T00:25:20.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827727+0000", 2020-03-09T00:25:20.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6050000000000005e-05 2020-03-09T00:25:20.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827803+0000", 2020-03-09T00:25:20.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99949370299999996 2020-03-09T00:25:20.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.097 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.097 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.097 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:398 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~425984] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:20.097 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.180531842000001, 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99725236100000003, 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.098 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 398 2020-03-09T00:25:20.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.099 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:20.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051814600000000004 2020-03-09T00:25:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831322+0000", 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0118e-05 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831333+0000", 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0710999999999999e-05 2020-03-09T00:25:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.102 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831353+0000", 2020-03-09T00:25:20.102 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7870000000000004e-06 2020-03-09T00:25:20.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831360+0000", 2020-03-09T00:25:20.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013810100000000001 2020-03-09T00:25:20.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831498+0000", 2020-03-09T00:25:20.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3380000000000001e-06 2020-03-09T00:25:20.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:20.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831502+0000", 2020-03-09T00:25:20.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97707543200000002 2020-03-09T00:25:20.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.808577+0000", 2020-03-09T00:25:20.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133669 2020-03-09T00:25:20.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.808711+0000", 2020-03-09T00:25:20.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010978260000000001 2020-03-09T00:25:20.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.809808+0000", 2020-03-09T00:25:20.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074316110000000003 2020-03-09T00:25:20.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817240+0000", 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010739297 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827979+0000", 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7324999999999996e-05 2020-03-09T00:25:20.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828057+0000", 2020-03-09T00:25:20.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99725236100000003 2020-03-09T00:25:20.109 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.109 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.109 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:399 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~344064] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.178430668000001, 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99540426400000004, 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.110 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.111 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.111 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.111 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.111 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 399 2020-03-09T00:25:20.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.111 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.112 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.112 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:20.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.112 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:20.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00031358200000000002 2020-03-09T00:25:20.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833219+0000", 2020-03-09T00:25:20.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4809999999999995e-06 2020-03-09T00:25:20.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833229+0000", 2020-03-09T00:25:20.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9320000000000008e-06 2020-03-09T00:25:20.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833238+0000", 2020-03-09T00:25:20.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0749999999999999e-06 2020-03-09T00:25:20.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833243+0000", 2020-03-09T00:25:20.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1554000000000001e-05 2020-03-09T00:25:20.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833294+0000", 2020-03-09T00:25:20.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4169999999999999e-06 2020-03-09T00:25:20.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833297+0000", 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.979004659 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812301+0000", 2020-03-09T00:25:20.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013289199999999999 2020-03-09T00:25:20.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812434+0000", 2020-03-09T00:25:20.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098429000000000008 2020-03-09T00:25:20.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813418+0000", 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0052416019999999997 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.818660+0000", 2020-03-09T00:25:20.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095537799999999996 2020-03-09T00:25:20.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.121 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.121 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828214+0000", 2020-03-09T00:25:20.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6000000000000002e-05 2020-03-09T00:25:20.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.122 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.122 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828310+0000", 2020-03-09T00:25:20.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99540426400000004 2020-03-09T00:25:20.122 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.122 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.122 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.122 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:400 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.178057067000001, 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99532314300000002, 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.123 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.124 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.124 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.125 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 400 2020-03-09T00:25:20.125 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.125 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:20.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9441999999999999e-05 2020-03-09T00:25:20.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833319+0000", 2020-03-09T00:25:20.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.421e-06 2020-03-09T00:25:20.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833325+0000", 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0309999999999998e-06 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833333+0000", 2020-03-09T00:25:20.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6189999999999998e-06 2020-03-09T00:25:20.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833336+0000", 2020-03-09T00:25:20.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5015999999999998e-05 2020-03-09T00:25:20.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833371+0000", 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.305e-06 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833372+0000", 2020-03-09T00:25:20.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.98258547500000004 2020-03-09T00:25:20.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.131 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.131 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.815957+0000", 2020-03-09T00:25:20.131 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000143335 2020-03-09T00:25:20.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816101+0000", 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00076297400000000003 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816864+0000", 2020-03-09T00:25:20.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001798628 2020-03-09T00:25:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.818662+0000", 2020-03-09T00:25:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0098116109999999996 2020-03-09T00:25:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828474+0000", 2020-03-09T00:25:20.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000128286 2020-03-09T00:25:20.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828602+0000", 2020-03-09T00:25:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99532314300000002 2020-03-09T00:25:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.135 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:401 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.177976040000001, 2020-03-09T00:25:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99818093799999996, 2020-03-09T00:25:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.136 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 401 2020-03-09T00:25:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.137 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3272e-05 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.139 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.139 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833383+0000", 2020-03-09T00:25:20.139 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2030000000000001e-06 2020-03-09T00:25:20.139 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.139 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.139 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.139 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833387+0000", 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0190000000000006e-06 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833395+0000", 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3800000000000001e-06 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.140 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833397+0000", 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9891000000000001e-05 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833437+0000", 2020-03-09T00:25:20.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.564e-06 2020-03-09T00:25:20.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.142 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:20.142 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833438+0000", 2020-03-09T00:25:20.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.98366688700000005 2020-03-09T00:25:20.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.143 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817105+0000", 2020-03-09T00:25:20.143 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000121193 2020-03-09T00:25:20.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:20.143 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817227+0000", 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011959001 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.829186+0000", 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012604700000000001 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.829312+0000", 2020-03-09T00:25:20.145 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022294810000000002 2020-03-09T00:25:20.145 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.145 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.831541+0000", 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99818093799999996 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.9s2 58/32 ECSubWrite(tid=151, reqid=osd.12.0:151, at_version=58'71, trim_to=0'0, roll_forward_to=57'67))", 2020-03-09T00:25:20.146 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:07.809026+0000", 2020-03-09T00:25:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 12.202309981999999, 2020-03-09T00:25:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.01546201, 2020-03-09T00:25:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.147 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.148 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809026+0000", 2020-03-09T00:25:20.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.148 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.148 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.148 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.148 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809026+0000", 2020-03-09T00:25:20.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9580999999999998e-05 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809086+0000", 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1883e-05 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.149 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809108+0000", 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7643999999999999e-05 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809125+0000", 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4909999999999996e-06 2020-03-09T00:25:20.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.809131+0000", 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0085116619999999997 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.151 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.817642+0000", 2020-03-09T00:25:20.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1310000000000001e-05 2020-03-09T00:25:20.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.152 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.152 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.817664+0000", 2020-03-09T00:25:20.152 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067742870000000004 2020-03-09T00:25:20.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824438+0000", 2020-03-09T00:25:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.0151999999999997e-05 2020-03-09T00:25:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:07.824488+0000", 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.01546201 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.154 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 64/48 ECSubWrite(tid=176, reqid=client.4751.0:544, at_version=64'78, trim_to=0'0, roll_forward_to=63'76))", 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:13.793965+0000", 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.2173715920000001, 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015769041000000001, 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.155 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.156 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.793965+0000", 2020-03-09T00:25:20.156 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.156 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.793965+0000", 2020-03-09T00:25:20.156 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00035591799999999999 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.794320+0000", 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4352e-05 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.157 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.158 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.794345+0000", 2020-03-09T00:25:20.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6469000000000001e-05 2020-03-09T00:25:20.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.158 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.158 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.794361+0000", 2020-03-09T00:25:20.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6290000000000004e-06 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.794368+0000", 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7992999999999999e-05 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.794426+0000", 2020-03-09T00:25:20.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0355e-05 2020-03-09T00:25:20.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.794436+0000", 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015247752999999999 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.809684+0000", 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9571999999999999e-05 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.809734+0000", 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015769041000000001 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.162 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.163 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 64/48 ECSubWrite(tid=177, reqid=client.4751.0:545, at_version=64'79, trim_to=0'0, roll_forward_to=63'76))", 2020-03-09T00:25:20.163 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:13.795053+0000", 2020-03-09T00:25:20.163 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 6.2162836639999997, 2020-03-09T00:25:20.163 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015489209, 2020-03-09T00:25:20.163 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.163 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:20.163 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.795053+0000", 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.795053+0000", 2020-03-09T00:25:20.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029380999999999998 2020-03-09T00:25:20.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.795346+0000", 2020-03-09T00:25:20.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0482000000000001e-05 2020-03-09T00:25:20.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.795367+0000", 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6538e-05 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.795383+0000", 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1400000000000002e-06 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.795390+0000", 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055062499999999999 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.800897+0000", 2020-03-09T00:25:20.168 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3689999999999999e-05 2020-03-09T00:25:20.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.800910+0000", 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095844539999999992 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.810495+0000", 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6845000000000003e-05 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:13.810542+0000", 2020-03-09T00:25:20.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015489209 2020-03-09T00:25:20.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.170 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.170 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs2 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.2078771259999996, 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019419130999999999, 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:20.171 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:20.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0024772549999999998 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805936+0000", 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6125000000000001e-05 2020-03-09T00:25:20.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805962+0000", 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5769000000000001e-05 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.174 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805978+0000", 2020-03-09T00:25:20.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.105e-06 2020-03-09T00:25:20.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805984+0000", 2020-03-09T00:25:20.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7604e-05 2020-03-09T00:25:20.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.806032+0000", 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2700999999999999e-05 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.806045+0000", 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016799584999999999 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.822844+0000", 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3986999999999998e-05 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.822878+0000", 2020-03-09T00:25:20.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019419130999999999 2020-03-09T00:25:20.178 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.178 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.178 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:564 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~507904] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.071824012, 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016199960999999999, 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.179 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.180 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.180 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.180 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 564 2020-03-09T00:25:20.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.180 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00049015199999999999 2020-03-09T00:25:20.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940002+0000", 2020-03-09T00:25:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0630000000000006e-06 2020-03-09T00:25:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940010+0000", 2020-03-09T00:25:20.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5739999999999998e-05 2020-03-09T00:25:20.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940026+0000", 2020-03-09T00:25:20.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8399999999999997e-06 2020-03-09T00:25:20.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940033+0000", 2020-03-09T00:25:20.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035392219999999999 2020-03-09T00:25:20.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.185 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.185 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943572+0000", 2020-03-09T00:25:20.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8063e-05 2020-03-09T00:25:20.185 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.185 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943660+0000", 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00097790300000000002 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944638+0000", 2020-03-09T00:25:20.186 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039951850000000001 2020-03-09T00:25:20.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.187 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.187 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948633+0000", 2020-03-09T00:25:20.187 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067926059999999996 2020-03-09T00:25:20.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955426+0000", 2020-03-09T00:25:20.188 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00028618699999999998 2020-03-09T00:25:20.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.188 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955712+0000", 2020-03-09T00:25:20.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016199960999999999 2020-03-09T00:25:20.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.189 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:565 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~622592] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.0676810670000001, 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016407618999999998, 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.190 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.191 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.191 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 565 2020-03-09T00:25:20.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.191 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00060946899999999996 2020-03-09T00:25:20.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944265+0000", 2020-03-09T00:25:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.004e-05 2020-03-09T00:25:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944275+0000", 2020-03-09T00:25:20.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7229000000000001e-05 2020-03-09T00:25:20.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944292+0000", 2020-03-09T00:25:20.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4939999999999997e-06 2020-03-09T00:25:20.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944299+0000", 2020-03-09T00:25:20.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046037999999999999 2020-03-09T00:25:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948903+0000", 2020-03-09T00:25:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000139479 2020-03-09T00:25:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.949043+0000", 2020-03-09T00:25:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011480489999999999 2020-03-09T00:25:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950191+0000", 2020-03-09T00:25:20.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055348039999999999 2020-03-09T00:25:20.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955725+0000", 2020-03-09T00:25:20.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041247109999999997 2020-03-09T00:25:20.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.959850+0000", 2020-03-09T00:25:20.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000212544 2020-03-09T00:25:20.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960063+0000", 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016407618999999998 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.201 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:566 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~32768] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:20.202 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:20.202 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.0669717529999998, 2020-03-09T00:25:20.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016519618999999999, 2020-03-09T00:25:20.202 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.202 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.202 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.203 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.203 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.203 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 566 2020-03-09T00:25:20.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.203 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:20.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5429e-05 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944430+0000", 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2100000000000003e-06 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.205 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944434+0000", 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4159999999999998e-06 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944441+0000", 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3349999999999998e-06 2020-03-09T00:25:20.206 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944445+0000", 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0093476589999999995 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953792+0000", 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014473700000000001 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953937+0000", 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00083984299999999995 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.954777+0000", 2020-03-09T00:25:20.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002244401 2020-03-09T00:25:20.209 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.209 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957021+0000", 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035682729999999998 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960590+0000", 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029431599999999998 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960884+0000", 2020-03-09T00:25:20.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016519618999999999 2020-03-09T00:25:20.211 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.211 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.212 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.212 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:567 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:20.212 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:20.213 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.0668620469999999, 2020-03-09T00:25:20.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017056153000000001, 2020-03-09T00:25:20.213 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.213 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:20.213 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.214 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.214 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.214 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 567 2020-03-09T00:25:20.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.214 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:20.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:20.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8381999999999999e-05 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944513+0000", 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1709999999999998e-06 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.216 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944516+0000", 2020-03-09T00:25:20.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7849999999999992e-06 2020-03-09T00:25:20.217 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.217 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.217 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.217 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944523+0000", 2020-03-09T00:25:20.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2550000000000002e-06 2020-03-09T00:25:20.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944527+0000", 2020-03-09T00:25:20.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011392539 2020-03-09T00:25:20.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955919+0000", 2020-03-09T00:25:20.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013716499999999999 2020-03-09T00:25:20.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956056+0000", 2020-03-09T00:25:20.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00057513499999999995 2020-03-09T00:25:20.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956632+0000", 2020-03-09T00:25:20.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00045027700000000001 2020-03-09T00:25:20.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957082+0000", 2020-03-09T00:25:20.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042876219999999996 2020-03-09T00:25:20.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:20.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.961369+0000", 2020-03-09T00:25:20.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000160822 2020-03-09T00:25:20.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.961530+0000", 2020-03-09T00:25:20.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017056153000000001 2020-03-09T00:25:20.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.224 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.224 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.224 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:568 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:20.224 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:20.225 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.0667685759999999, 2020-03-09T00:25:20.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020843882000000001, 2020-03-09T00:25:20.225 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.225 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:20.225 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:20.225 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:20.226 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:20.226 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 568 2020-03-09T00:25:20.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.226 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.226 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:20.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.227 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.227 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.227 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:20.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8635999999999999e-05 2020-03-09T00:25:20.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944626+0000", 2020-03-09T00:25:20.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2609999999999999e-06 2020-03-09T00:25:20.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944632+0000", 2020-03-09T00:25:20.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4993e-05 2020-03-09T00:25:20.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944647+0000", 2020-03-09T00:25:20.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9439999999999996e-06 2020-03-09T00:25:20.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944652+0000", 2020-03-09T00:25:20.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013280882000000001 2020-03-09T00:25:20.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957933+0000", 2020-03-09T00:25:20.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012477000000000001 2020-03-09T00:25:20.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:20.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.958058+0000", 2020-03-09T00:25:20.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0044866419999999999 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962545+0000", 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000120438 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:20.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962665+0000", 2020-03-09T00:25:20.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027463159999999999 2020-03-09T00:25:20.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.965411+0000", 2020-03-09T00:25:20.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020843882000000001 2020-03-09T00:25:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs2 68/67 ECSubWrite(tid=202, reqid=client.4751.0:572, at_version=68'64, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.948221+0000", 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 2.0631147489999999, 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014738256, 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:20.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948221+0000", 2020-03-09T00:25:20.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:20.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:20.238 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948221+0000", 2020-03-09T00:25:20.238 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024302199999999999 2020-03-09T00:25:20.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:20.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948464+0000", 2020-03-09T00:25:20.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027889910000000002 2020-03-09T00:25:20.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.951253+0000", 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5315000000000002e-05 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.951269+0000", 2020-03-09T00:25:20.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.544e-06 2020-03-09T00:25:20.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:20.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.951273+0000", 2020-03-09T00:25:20.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7008999999999997e-05 2020-03-09T00:25:20.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:20.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.951320+0000", 2020-03-09T00:25:20.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2576e-05 2020-03-09T00:25:20.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:20.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.951333+0000", 2020-03-09T00:25:20.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011585450000000001 2020-03-09T00:25:20.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:20.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962918+0000", 2020-03-09T00:25:20.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.1349e-05 2020-03-09T00:25:20.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:20.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:20.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:20.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962960+0000", 2020-03-09T00:25:20.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014738256 2020-03-09T00:25:20.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.246 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.247 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:20.247 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:20.247 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:20.251 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:20.251 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:20.251 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:20.251 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:20.246+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:20.252 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:25:20.261 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:20.281 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:20.302 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:20.324 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:20.324 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:20.322+0000 7fdf12ffd700 0 -- 172.21.15.148:0/1275615168 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7fdf14187e90 msgr2=0x7fdf1418a280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:20.349 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:20.347+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:20.389 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:20.389 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:20.398 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:25:20.450 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:20.448+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:20.454 INFO:teuthology.orchestra.run.smithi139.stdout: marking collection for removal 2020-03-09T00:25:20.455 INFO:teuthology.orchestra.run.smithi139.stdout:setting '_remove' omap key 2020-03-09T00:25:20.455 INFO:teuthology.orchestra.run.smithi139.stdout:finish_remove_pgs 2.7_head removing 2.7 2020-03-09T00:25:20.458 INFO:teuthology.orchestra.run.smithi139.stdout:Remove successful 2020-03-09T00:25:20.523 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:20.523 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:20.532 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:25:20.545 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:20.547 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.547 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.547 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.547 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.547 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.547 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.548 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:20.548 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:20.548 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.548 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.548 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.548 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:20.552 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:20.550+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:20.557 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:20.652 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:20.650+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:20.656 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:20.657 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:20.665 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:25:20.753 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:20.751+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:20.784 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:20.784 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:20.793 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:25:20.798 INFO:tasks.rados.rados.0.smithi104.stdout:239: done (0 left) 2020-03-09T00:25:20.798 INFO:tasks.rados.rados.0.smithi104.stdout:240: rollback oid 16 current snap is 19 2020-03-09T00:25:20.798 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 16 to 16 2020-03-09T00:25:20.799 INFO:tasks.rados.rados.0.smithi104.stdout:241: read oid 7 snap 16 2020-03-09T00:25:20.799 INFO:tasks.rados.rados.0.smithi104.stdout:241: expect (ObjNum 97 snap 16 seq_num 97) 2020-03-09T00:25:20.799 INFO:tasks.rados.rados.0.smithi104.stdout:242: snap_create 2020-03-09T00:25:20.808 INFO:tasks.rados.rados.0.smithi104.stdout:240: finishing rollback tid 0 to smithi10428818-16 2020-03-09T00:25:20.810 INFO:tasks.rados.rados.0.smithi104.stdout:240: finishing rollback tid 1 to smithi10428818-16 2020-03-09T00:25:20.821 INFO:tasks.rados.rados.0.smithi104.stdout:241: expect (ObjNum 97 snap 16 seq_num 97) 2020-03-09T00:25:20.854 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:20.852+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:20.864 INFO:tasks.rados.rados.0.smithi104.stdout:240: finishing rollback tid 2 to smithi10428818-16 2020-03-09T00:25:20.864 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 16 v 139 (ObjNum 70 snap 11 seq_num 70) dirty exists 2020-03-09T00:25:20.914 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:20.914 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:20.922 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:25:20.955 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:20.953+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:20.961 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-03-09T00:25:20.972 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.7.6 2020-03-09T00:25:21.045 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.045 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.054 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:25:21.056 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:21.054+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:21.158 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:21.155+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:21.174 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.174 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.182 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:25:21.261 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:21.257+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:21.296 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.297 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.306 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:25:21.360 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:21.358+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:21.421 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.421 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.429 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:25:21.461 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:21.459+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:21.547 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.547 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.556 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:25:21.562 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:21.560+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:21.665 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:21.661+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:21.672 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.672 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.680 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:25:21.763 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:21.761+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:21.769 INFO:tasks.rados.rados.0.smithi104.stdout:240: done (2 left) 2020-03-09T00:25:21.769 INFO:tasks.rados.rados.0.smithi104.stdout:241: done (1 left) 2020-03-09T00:25:21.769 INFO:tasks.rados.rados.0.smithi104.stdout:242: done (0 left) 2020-03-09T00:25:21.769 INFO:tasks.rados.rados.0.smithi104.stdout:243: read oid 41 snap 18 2020-03-09T00:25:21.769 INFO:tasks.rados.rados.0.smithi104.stdout:243: expect (ObjNum 40 snap 0 seq_num 40) 2020-03-09T00:25:21.769 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 3 current snap is 20 2020-03-09T00:25:21.769 INFO:tasks.rados.rados.0.smithi104.stdout:244: seq_num 107 ranges {0=638976,638976=458752,1097728=704512} 2020-03-09T00:25:21.770 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is unset 2020-03-09T00:25:21.771 INFO:tasks.rados.rados.0.smithi104.stdout:244: writing smithi10428818-3 from 0 to 638976 tid 1 2020-03-09T00:25:21.774 INFO:tasks.rados.rados.0.smithi104.stdout:244: writing smithi10428818-3 from 638976 to 1097728 tid 2 2020-03-09T00:25:21.779 INFO:tasks.rados.rados.0.smithi104.stdout:244: writing smithi10428818-3 from 1097728 to 1802240 tid 3 2020-03-09T00:25:21.779 INFO:tasks.rados.rados.0.smithi104.stdout:245: snap_remove snap 16 2020-03-09T00:25:21.800 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.800 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.809 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:25:21.821 INFO:teuthology.orchestra.run.smithi139.stdout:get_pg_num_history pg_num_history pg_num_history(e66 pg_nums {1={3=1},2={27=8},3={32=16}} deleted_pools ) 2020-03-09T00:25:21.821 INFO:teuthology.orchestra.run.smithi139.stdout:Importing pgid 2.7 2020-03-09T00:25:21.821 INFO:teuthology.orchestra.run.smithi139.stdout:write_pg epoch 46 info 2.7( empty local-lis/les=45/46 n=0 ec=27/27 lis/c=45/45 les/c/f=46/46/0 sis=45) 2020-03-09T00:25:21.822 INFO:teuthology.orchestra.run.smithi139.stdout:Import successful 2020-03-09T00:25:21.864 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:21.862+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:21.947 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:21.947 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:21.952 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:25:21.965 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:21.963+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:22.066 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:22.065+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:22.116 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:22.117 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:22.125 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:25:22.167 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:22.165+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:22.251 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:22.251 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:22.260 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:25:22.267 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:22.266+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:22.343 INFO:teuthology.orchestra.run.smithi139:> sudo rm -f /var/log/ceph/exp.2.7.6 2020-03-09T00:25:22.368 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:22.366+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:22.387 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:22.387 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:22.396 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:25:22.412 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:25:22.482 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:22.480+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:22.517 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:22.517 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:22.526 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:25:22.557 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:22.569 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:22.567+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:22.670 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:22.667+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:22.688 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:22.688 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:22.698 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:25:22.710 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:22.726 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:25:22.728 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":72,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:21.761608+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-03-09T00:24:40.793412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":36,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":21,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:25:22.732 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:22.738 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 1, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:09.667386+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 1, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 1, u'pg_num': 1, u'type': 1, u'grade_table': [], u'pool_name': u'device_health_metrics', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 1, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'mgr_devicehealth': {}}, u'options': {u'pg_num_min': 1}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'26', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:26.419136+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'29', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 16384, u'flags_names': u'hashpspool,selfmanaged_snaps', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 16, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-09T00:24:40.793412+0000', u'snap_epoch': 72, u'erasure_code_profile': u'shecprofile', u'pg_autoscale_mode': u'off', u'snap_seq': 36, u'expected_num_objects': 0, u'size': 7, u'pg_num_pending': 16, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 16, u'pg_num': 16, u'type': 3, u'grade_table': [], u'pool_name': u'unique_pool_0', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 16, u'cache_mode': u'none', u'min_size': 5, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rados': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 8193, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'72', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-03-09T00:25:22.738 INFO:teuthology.orchestra.run.smithi139:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op apply-layout-settings --pool device_health_metrics 2020-03-09T00:25:22.753 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:22.772 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:22.770+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:22.774 INFO:tasks.rados.rados.0.smithi104.stdout:245: done (2 left) 2020-03-09T00:25:22.774 INFO:tasks.rados.rados.0.smithi104.stdout:246: snap_create 2020-03-09T00:25:22.774 INFO:tasks.rados.rados.0.smithi104.stdout:243: expect (ObjNum 40 snap 0 seq_num 40) 2020-03-09T00:25:22.774 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:22.792 INFO:tasks.rados.rados.0.smithi104.stdout:244: finishing write tid 1 to smithi10428818-3 2020-03-09T00:25:22.792 INFO:tasks.rados.rados.0.smithi104.stdout:244: finishing write tid 2 to smithi10428818-3 2020-03-09T00:25:22.792 INFO:tasks.rados.rados.0.smithi104.stdout:244: finishing write tid 3 to smithi10428818-3 2020-03-09T00:25:22.792 INFO:tasks.rados.rados.0.smithi104.stdout:244: finishing write tid 5 to smithi10428818-3 2020-03-09T00:25:22.792 INFO:tasks.rados.rados.0.smithi104.stdout:244: finishing write tid 6 to smithi10428818-3 2020-03-09T00:25:22.792 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 3 v 77 (ObjNum 107 snap 20 seq_num 107) dirty exists 2020-03-09T00:25:22.793 INFO:tasks.rados.rados.0.smithi104.stdout:244: left oid 3 (ObjNum 107 snap 20 seq_num 107) 2020-03-09T00:25:22.794 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:22.826 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:22.826 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:22.827 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:22.829 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:25:22.836 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:22.855 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:22.874 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:22.871+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:22.875 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:22.896 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:22.916 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:22.935 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:22.948 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:22.948 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:22.949 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:22.949 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:22.949 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:22.957 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:22.958 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:25:22.977 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:22.996 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:23.008 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:23.000+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:23.008 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:23.075 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:23.075 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:23.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:23.076 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:23.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.076 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:23.076 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:23.076 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.245403723999999, 2020-03-09T00:25:23.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737, 2020-03-09T00:25:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000156164 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827665+0000", 2020-03-09T00:25:23.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5397000000000001e-05 2020-03-09T00:25:23.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827690+0000", 2020-03-09T00:25:23.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6900000000000002e-05 2020-03-09T00:25:23.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827727+0000", 2020-03-09T00:25:23.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9529999999999999e-06 2020-03-09T00:25:23.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827735+0000", 2020-03-09T00:25:23.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011533660000000001 2020-03-09T00:25:23.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828889+0000", 2020-03-09T00:25:23.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1542999999999999e-05 2020-03-09T00:25:23.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828900+0000", 2020-03-09T00:25:23.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038471730000000002 2020-03-09T00:25:23.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832747+0000", 2020-03-09T00:25:23.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8874000000000001e-05 2020-03-09T00:25:23.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832786+0000", 2020-03-09T00:25:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737 2020-03-09T00:25:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:25:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.208806295999999, 2020-03-09T00:25:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045310819999999996, 2020-03-09T00:25:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.085 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000157849 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.087 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864264+0000", 2020-03-09T00:25:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8293000000000002e-05 2020-03-09T00:25:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864292+0000", 2020-03-09T00:25:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6464e-05 2020-03-09T00:25:23.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864309+0000", 2020-03-09T00:25:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9360000000000002e-06 2020-03-09T00:25:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864316+0000", 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3850000000000001e-05 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864370+0000", 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3759e-05 2020-03-09T00:25:23.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864383+0000", 2020-03-09T00:25:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041983749999999999 2020-03-09T00:25:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868582+0000", 2020-03-09T00:25:23.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5556000000000001e-05 2020-03-09T00:25:23.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868637+0000", 2020-03-09T00:25:23.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045310819999999996 2020-03-09T00:25:23.093 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.093 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.093 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.094 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 44/32 ECSubWrite(tid=34, reqid=client.4751.0:249, at_version=44'16, trim_to=0'0, roll_forward_to=44'14))", 2020-03-09T00:25:23.094 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.867338+0000", 2020-03-09T00:25:23.094 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.205574901999999, 2020-03-09T00:25:23.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038215200000000001, 2020-03-09T00:25:23.094 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.095 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.095 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.095 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.095 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.095 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867338+0000", 2020-03-09T00:25:23.095 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867338+0000", 2020-03-09T00:25:23.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027359900000000002 2020-03-09T00:25:23.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867611+0000", 2020-03-09T00:25:23.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4257000000000001e-05 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867646+0000", 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1376e-05 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.099 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867667+0000", 2020-03-09T00:25:23.099 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1700000000000003e-06 2020-03-09T00:25:23.099 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.100 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867676+0000", 2020-03-09T00:25:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2354999999999995e-05 2020-03-09T00:25:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867738+0000", 2020-03-09T00:25:23.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6375000000000002e-05 2020-03-09T00:25:23.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.867755+0000", 2020-03-09T00:25:23.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033428120000000001 2020-03-09T00:25:23.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.871098+0000", 2020-03-09T00:25:23.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1575999999999995e-05 2020-03-09T00:25:23.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.871159+0000", 2020-03-09T00:25:23.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038215200000000001 2020-03-09T00:25:23.105 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.105 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.105 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.105 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s1 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:23.105 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:23.106 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.280125331000001, 2020-03-09T00:25:23.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997, 2020-03-09T00:25:23.106 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.106 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.106 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.106 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.107 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.107 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:23.107 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.107 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:23.108 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062438599999999999 2020-03-09T00:25:23.108 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.108 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.108 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.108 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793412+0000", 2020-03-09T00:25:23.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6639e-05 2020-03-09T00:25:23.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.109 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.109 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793438+0000", 2020-03-09T00:25:23.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3116999999999999e-05 2020-03-09T00:25:23.109 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.110 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793461+0000", 2020-03-09T00:25:23.110 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2469999999999997e-06 2020-03-09T00:25:23.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.110 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.110 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793471+0000", 2020-03-09T00:25:23.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9004999999999994e-05 2020-03-09T00:25:23.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793540+0000", 2020-03-09T00:25:23.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8408000000000001e-05 2020-03-09T00:25:23.112 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.112 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.112 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.112 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793558+0000", 2020-03-09T00:25:23.112 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053623589999999997 2020-03-09T00:25:23.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.113 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798920+0000", 2020-03-09T00:25:23.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2633000000000002e-05 2020-03-09T00:25:23.114 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.114 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798973+0000", 2020-03-09T00:25:23.114 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997 2020-03-09T00:25:23.114 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.115 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.115 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.115 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.115 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=30, reqid=client.4751.0:346, at_version=53'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:23.115 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:23.115 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.245867269000001, 2020-03-09T00:25:23.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999, 2020-03-09T00:25:23.116 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.116 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.116 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:23.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:23.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013296799999999999 2020-03-09T00:25:23.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827178+0000", 2020-03-09T00:25:23.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6305e-05 2020-03-09T00:25:23.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827205+0000", 2020-03-09T00:25:23.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5832999999999999e-05 2020-03-09T00:25:23.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827220+0000", 2020-03-09T00:25:23.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7170000000000008e-06 2020-03-09T00:25:23.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827228+0000", 2020-03-09T00:25:23.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2020-03-09T00:25:23.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827292+0000", 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4157e-05 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827306+0000", 2020-03-09T00:25:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046893439999999998 2020-03-09T00:25:23.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831995+0000", 2020-03-09T00:25:23.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3732000000000006e-05 2020-03-09T00:25:23.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.123 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.123 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832059+0000", 2020-03-09T00:25:23.123 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999 2020-03-09T00:25:23.123 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.123 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.123 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.124 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.124 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.124 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=31, reqid=client.4751.0:347, at_version=53'19, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:23.124 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:23.124 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.238029129000001, 2020-03-09T00:25:23.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996, 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:23.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030527400000000002 2020-03-09T00:25:23.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835189+0000", 2020-03-09T00:25:23.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4273999999999998e-05 2020-03-09T00:25:23.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835213+0000", 2020-03-09T00:25:23.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7130999999999999e-05 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835230+0000", 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6400000000000003e-06 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835239+0000", 2020-03-09T00:25:23.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0665000000000003e-05 2020-03-09T00:25:23.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835299+0000", 2020-03-09T00:25:23.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6062e-05 2020-03-09T00:25:23.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835315+0000", 2020-03-09T00:25:23.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005161349 2020-03-09T00:25:23.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840477+0000", 2020-03-09T00:25:23.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7466000000000003e-05 2020-03-09T00:25:23.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840524+0000", 2020-03-09T00:25:23.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996 2020-03-09T00:25:23.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.132 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.132 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.133 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=34, reqid=client.4751.0:349, at_version=0'0, trim_to=0'0, roll_forward_to=53'21))", 2020-03-09T00:25:23.133 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:23.133 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.227451079000001, 2020-03-09T00:25:23.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001, 2020-03-09T00:25:23.133 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.134 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.134 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:23.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:23.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6162000000000002e-05 2020-03-09T00:25:23.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845518+0000", 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3681e-05 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845531+0000", 2020-03-09T00:25:23.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376e-05 2020-03-09T00:25:23.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.137 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.137 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845547+0000", 2020-03-09T00:25:23.137 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8040000000000002e-06 2020-03-09T00:25:23.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.138 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845555+0000", 2020-03-09T00:25:23.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010135 2020-03-09T00:25:23.138 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.138 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.138 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845656+0000", 2020-03-09T00:25:23.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2714000000000001e-05 2020-03-09T00:25:23.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845669+0000", 2020-03-09T00:25:23.139 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072973140000000001 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.852966+0000", 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2704999999999998e-05 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.853019+0000", 2020-03-09T00:25:23.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001 2020-03-09T00:25:23.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.141 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.141 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.142 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 59/32 ECSubWrite(tid=76, reqid=osd.10.0:76, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:23.142 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.822198+0000", 2020-03-09T00:25:23.142 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 14.250714347000001, 2020-03-09T00:25:23.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042710029999999998, 2020-03-09T00:25:23.142 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.142 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.142 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822198+0000", 2020-03-09T00:25:23.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822198+0000", 2020-03-09T00:25:23.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0723000000000001e-05 2020-03-09T00:25:23.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822259+0000", 2020-03-09T00:25:23.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2785999999999998e-05 2020-03-09T00:25:23.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822282+0000", 2020-03-09T00:25:23.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4289e-05 2020-03-09T00:25:23.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.146 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822296+0000", 2020-03-09T00:25:23.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023975810000000002 2020-03-09T00:25:23.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.146 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.824694+0000", 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00065045999999999995 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825344+0000", 2020-03-09T00:25:23.147 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4735e-05 2020-03-09T00:25:23.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.148 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.148 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825359+0000", 2020-03-09T00:25:23.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010596990000000001 2020-03-09T00:25:23.148 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.148 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826418+0000", 2020-03-09T00:25:23.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0729999999999997e-05 2020-03-09T00:25:23.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826469+0000", 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042710029999999998 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:23.150 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:23.151 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 14.158932241, 2020-03-09T00:25:23.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014134343000000001, 2020-03-09T00:25:23.151 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.151 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.151 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:23.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:23.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00044913599999999999 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914429+0000", 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3470000000000002e-05 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.153 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.154 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914473+0000", 2020-03-09T00:25:23.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4422000000000001e-05 2020-03-09T00:25:23.154 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.154 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914497+0000", 2020-03-09T00:25:23.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4169999999999997e-06 2020-03-09T00:25:23.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914504+0000", 2020-03-09T00:25:23.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3522999999999995e-05 2020-03-09T00:25:23.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914567+0000", 2020-03-09T00:25:23.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4911e-05 2020-03-09T00:25:23.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914582+0000", 2020-03-09T00:25:23.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013495323 2020-03-09T00:25:23.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.928078+0000", 2020-03-09T00:25:23.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7141000000000002e-05 2020-03-09T00:25:23.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.928115+0000", 2020-03-09T00:25:23.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014134343000000001 2020-03-09T00:25:23.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.160 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.160 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.161 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 63/32 ECSubWrite(tid=83, reqid=osd.10.0:83, at_version=63'49, trim_to=0'0, roll_forward_to=60'47))", 2020-03-09T00:25:23.161 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:23.161 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.297511662, 2020-03-09T00:25:23.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073598240000000001, 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:23.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4988000000000003e-05 2020-03-09T00:25:23.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775456+0000", 2020-03-09T00:25:23.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6618000000000002e-05 2020-03-09T00:25:23.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775483+0000", 2020-03-09T00:25:23.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5108e-05 2020-03-09T00:25:23.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.165 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.165 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775498+0000", 2020-03-09T00:25:23.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7239999999999992e-06 2020-03-09T00:25:23.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775505+0000", 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110979 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775616+0000", 2020-03-09T00:25:23.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2696e-05 2020-03-09T00:25:23.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775629+0000", 2020-03-09T00:25:23.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00708429 2020-03-09T00:25:23.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782713+0000", 2020-03-09T00:25:23.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7420999999999999e-05 2020-03-09T00:25:23.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782761+0000", 2020-03-09T00:25:23.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073598240000000001 2020-03-09T00:25:23.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.169 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.169 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.171 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 63/48 ECSubWrite(tid=42, reqid=osd.15.0:42, at_version=63'29, trim_to=0'0, roll_forward_to=59'27))", 2020-03-09T00:25:23.171 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:23.171 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.29673884, 2020-03-09T00:25:23.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005556643, 2020-03-09T00:25:23.171 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.171 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.172 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:23.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:23.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8814000000000003e-05 2020-03-09T00:25:23.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776233+0000", 2020-03-09T00:25:23.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2234000000000001e-05 2020-03-09T00:25:23.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776255+0000", 2020-03-09T00:25:23.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6149000000000001e-05 2020-03-09T00:25:23.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776271+0000", 2020-03-09T00:25:23.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0389999999999999e-06 2020-03-09T00:25:23.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.176 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776278+0000", 2020-03-09T00:25:23.176 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040392100000000001 2020-03-09T00:25:23.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.176 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776682+0000", 2020-03-09T00:25:23.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3020000000000001e-05 2020-03-09T00:25:23.177 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776695+0000", 2020-03-09T00:25:23.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049768449999999997 2020-03-09T00:25:23.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781672+0000", 2020-03-09T00:25:23.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8621e-05 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781730+0000", 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005556643 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.179 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.180 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 63/32 ECSubWrite(tid=84, reqid=osd.10.0:84, at_version=63'51, trim_to=0'0, roll_forward_to=60'47))", 2020-03-09T00:25:23.180 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:23.181 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.296262284000001, 2020-03-09T00:25:23.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061568409999999997, 2020-03-09T00:25:23.181 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.181 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.181 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:23.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:23.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2492999999999999e-05 2020-03-09T00:25:23.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776723+0000", 2020-03-09T00:25:23.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2741000000000001e-05 2020-03-09T00:25:23.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776746+0000", 2020-03-09T00:25:23.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5485000000000002e-05 2020-03-09T00:25:23.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776761+0000", 2020-03-09T00:25:23.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4679999999999997e-06 2020-03-09T00:25:23.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776767+0000", 2020-03-09T00:25:23.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012315000000000001 2020-03-09T00:25:23.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776891+0000", 2020-03-09T00:25:23.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2254e-05 2020-03-09T00:25:23.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776903+0000", 2020-03-09T00:25:23.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058656510000000004 2020-03-09T00:25:23.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782769+0000", 2020-03-09T00:25:23.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8599000000000001e-05 2020-03-09T00:25:23.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782807+0000", 2020-03-09T00:25:23.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061568409999999997 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 63/32 ECSubWrite(tid=14, reqid=osd.3.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=63'12))", 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.777416+0000", 2020-03-09T00:25:23.190 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.295496547999999, 2020-03-09T00:25:23.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044778049999999996, 2020-03-09T00:25:23.191 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.191 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.191 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777416+0000", 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777416+0000", 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3637999999999997e-05 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777470+0000", 2020-03-09T00:25:23.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1753000000000001e-05 2020-03-09T00:25:23.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777481+0000", 2020-03-09T00:25:23.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4347e-05 2020-03-09T00:25:23.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777496+0000", 2020-03-09T00:25:23.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7529999999999998e-06 2020-03-09T00:25:23.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777502+0000", 2020-03-09T00:25:23.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000196279 2020-03-09T00:25:23.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777698+0000", 2020-03-09T00:25:23.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1268e-05 2020-03-09T00:25:23.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777709+0000", 2020-03-09T00:25:23.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041370770000000003 2020-03-09T00:25:23.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781846+0000", 2020-03-09T00:25:23.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7689999999999999e-05 2020-03-09T00:25:23.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781894+0000", 2020-03-09T00:25:23.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044778049999999996 2020-03-09T00:25:23.198 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.198 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.199 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.199 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s1 63/54 ECSubWrite(tid=31, reqid=osd.0.0:31, at_version=63'42, trim_to=0'0, roll_forward_to=60'38))", 2020-03-09T00:25:23.199 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.781663+0000", 2020-03-09T00:25:23.199 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.291249603000001, 2020-03-09T00:25:23.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043990009999999996, 2020-03-09T00:25:23.200 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.200 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.200 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781663+0000", 2020-03-09T00:25:23.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781663+0000", 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8446000000000003e-05 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781721+0000", 2020-03-09T00:25:23.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.933e-05 2020-03-09T00:25:23.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781741+0000", 2020-03-09T00:25:23.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3501e-05 2020-03-09T00:25:23.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781754+0000", 2020-03-09T00:25:23.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5519999999999999e-06 2020-03-09T00:25:23.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781762+0000", 2020-03-09T00:25:23.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00059762200000000002 2020-03-09T00:25:23.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782359+0000", 2020-03-09T00:25:23.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2493999999999999e-05 2020-03-09T00:25:23.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782372+0000", 2020-03-09T00:25:23.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036379670000000002 2020-03-09T00:25:23.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.786010+0000", 2020-03-09T00:25:23.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2089000000000002e-05 2020-03-09T00:25:23.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.786062+0000", 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043990009999999996 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 63/48 ECSubWrite(tid=43, reqid=osd.15.0:42, at_version=0'0, trim_to=0'0, roll_forward_to=63'29))", 2020-03-09T00:25:23.209 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.783061+0000", 2020-03-09T00:25:23.210 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.289851923000001, 2020-03-09T00:25:23.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040721150000000003, 2020-03-09T00:25:23.210 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.210 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.210 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783061+0000", 2020-03-09T00:25:23.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783061+0000", 2020-03-09T00:25:23.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5686999999999997e-05 2020-03-09T00:25:23.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783116+0000", 2020-03-09T00:25:23.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.272e-05 2020-03-09T00:25:23.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783129+0000", 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5832999999999999e-05 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783145+0000", 2020-03-09T00:25:23.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3820000000000001e-06 2020-03-09T00:25:23.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783151+0000", 2020-03-09T00:25:23.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8148e-05 2020-03-09T00:25:23.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783199+0000", 2020-03-09T00:25:23.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2207e-05 2020-03-09T00:25:23.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.783212+0000", 2020-03-09T00:25:23.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003865729 2020-03-09T00:25:23.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.787077+0000", 2020-03-09T00:25:23.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5408999999999998e-05 2020-03-09T00:25:23.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.787133+0000", 2020-03-09T00:25:23.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040721150000000003 2020-03-09T00:25:23.217 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.217 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 68/67 ECSubWrite(tid=91, reqid=client.4751.0:580, at_version=68'59, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.1055945510000003, 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006533311, 2020-03-09T00:25:23.218 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.219 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.219 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:23.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:23.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089329700000000004 2020-03-09T00:25:23.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968211+0000", 2020-03-09T00:25:23.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4736000000000001e-05 2020-03-09T00:25:23.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968246+0000", 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0851000000000001e-05 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968267+0000", 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1970000000000001e-05 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968279+0000", 2020-03-09T00:25:23.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6097999999999998e-05 2020-03-09T00:25:23.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968325+0000", 2020-03-09T00:25:23.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2437e-05 2020-03-09T00:25:23.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968337+0000", 2020-03-09T00:25:23.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054777970000000004 2020-03-09T00:25:23.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.973815+0000", 2020-03-09T00:25:23.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6124999999999997e-05 2020-03-09T00:25:23.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.973851+0000", 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006533311 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.227 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 68/67 ECSubWrite(tid=101, reqid=client.4751.0:592, at_version=68'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:23.228 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:23.228 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.0838410920000001, 2020-03-09T00:25:23.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01056028, 2020-03-09T00:25:23.228 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.228 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.228 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:23.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00055521499999999996 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989627+0000", 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4139999999999999e-05 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989651+0000", 2020-03-09T00:25:23.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5061e-05 2020-03-09T00:25:23.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989666+0000", 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1540000000000002e-06 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989674+0000", 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9898000000000001e-05 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989734+0000", 2020-03-09T00:25:23.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5279000000000001e-05 2020-03-09T00:25:23.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989749+0000", 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098335769999999996 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999583+0000", 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8955999999999997e-05 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999632+0000", 2020-03-09T00:25:23.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01056028 2020-03-09T00:25:23.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.235 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.235 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 68/67 ECSubWrite(tid=102, reqid=client.4751.0:593, at_version=68'24, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.0798990990000004, 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066457310000000002, 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.236 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:23.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037136200000000003 2020-03-09T00:25:23.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993385+0000", 2020-03-09T00:25:23.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2940999999999999e-05 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993408+0000", 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376e-05 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993423+0000", 2020-03-09T00:25:23.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6939999999999995e-06 2020-03-09T00:25:23.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.240 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.240 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993431+0000", 2020-03-09T00:25:23.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9996e-05 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993481+0000", 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6351999999999999e-05 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993537+0000", 2020-03-09T00:25:23.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061008299999999998 2020-03-09T00:25:23.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999638+0000", 2020-03-09T00:25:23.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1180000000000001e-05 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999659+0000", 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066457310000000002 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.243 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.244 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 69/32 ECSubWrite(tid=20, reqid=client.4751.0:599, at_version=69'19, trim_to=0'0, roll_forward_to=67'16))", 2020-03-09T00:25:23.244 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:23.244 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2781739339999998, 2020-03-09T00:25:23.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056539340000000002, 2020-03-09T00:25:23.244 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:23.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1377000000000004e-05 2020-03-09T00:25:23.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794820+0000", 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3682000000000001e-05 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794844+0000", 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4918e-05 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794859+0000", 2020-03-09T00:25:23.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4799999999999998e-06 2020-03-09T00:25:23.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794865+0000", 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016106930000000001 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796476+0000", 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3885e-05 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796490+0000", 2020-03-09T00:25:23.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003855261 2020-03-09T00:25:23.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800345+0000", 2020-03-09T00:25:23.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7638e-05 2020-03-09T00:25:23.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800393+0000", 2020-03-09T00:25:23.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056539340000000002 2020-03-09T00:25:23.251 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.252 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.252 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.252 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 69/32 ECSubWrite(tid=21, reqid=client.4751.0:600, at_version=69'20, trim_to=0'0, roll_forward_to=67'16))", 2020-03-09T00:25:23.252 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.796264+0000", 2020-03-09T00:25:23.252 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.2766487120000001, 2020-03-09T00:25:23.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041635279999999997, 2020-03-09T00:25:23.253 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:23.253 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.253 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:23.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:23.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796264+0000", 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796264+0000", 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7522999999999998e-05 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:23.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796321+0000", 2020-03-09T00:25:23.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9924000000000001e-05 2020-03-09T00:25:23.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:23.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796341+0000", 2020-03-09T00:25:23.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5067e-05 2020-03-09T00:25:23.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:23.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796356+0000", 2020-03-09T00:25:23.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8300000000000002e-06 2020-03-09T00:25:23.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796365+0000", 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013635800000000001 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:23.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.797729+0000", 2020-03-09T00:25:23.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1219999999999999e-05 2020-03-09T00:25:23.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.797740+0000", 2020-03-09T00:25:23.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026556069999999999 2020-03-09T00:25:23.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800396+0000", 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1776999999999997e-05 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:23.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800427+0000", 2020-03-09T00:25:23.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041635279999999997 2020-03-09T00:25:23.260 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.260 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.260 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.260 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:23.260 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:23.260 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:23.262 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:23.259+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:23.262 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:23.363 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:23.360+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:23.400 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:23.400 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:23.400 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:23.400 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:23.409 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:23.493 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:23.491+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:23.564 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:23.562+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:23.574 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 303, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 274, in copy_to_log capture.write(line) TypeError: unbound method write() must be called with StringIO instance as first argument (got str instance instead) 2020-03-09T00:25:23.574 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:23.575 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:23.575 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:23.575 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:23.575 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:23.582 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:23.665 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:23.664+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:23.748 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:23.749 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:23.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:23.749 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:23.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.750 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:23.750 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:23.750 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 29.891520298, 2020-03-09T00:25:23.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010037961, 2020-03-09T00:25:23.750 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.750 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040949200000000002 2020-03-09T00:25:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854532+0000", 2020-03-09T00:25:23.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1631000000000003e-05 2020-03-09T00:25:23.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854564+0000", 2020-03-09T00:25:23.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2872000000000001e-05 2020-03-09T00:25:23.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854587+0000", 2020-03-09T00:25:23.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0699999999999999e-05 2020-03-09T00:25:23.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854598+0000", 2020-03-09T00:25:23.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6437000000000001e-05 2020-03-09T00:25:23.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854664+0000", 2020-03-09T00:25:23.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6180999999999998e-05 2020-03-09T00:25:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854700+0000", 2020-03-09T00:25:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094130399999999993 2020-03-09T00:25:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864113+0000", 2020-03-09T00:25:23.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7608000000000002e-05 2020-03-09T00:25:23.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864161+0000", 2020-03-09T00:25:23.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010037961 2020-03-09T00:25:23.758 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.758 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.758 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.759 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:23.759 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.856927+0000", 2020-03-09T00:25:23.759 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 29.888715735000002, 2020-03-09T00:25:23.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072822809999999998, 2020-03-09T00:25:23.759 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.760 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.760 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.760 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856927+0000", 2020-03-09T00:25:23.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.761 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.761 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856927+0000", 2020-03-09T00:25:23.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014181799999999999 2020-03-09T00:25:23.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857069+0000", 2020-03-09T00:25:23.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7234000000000001e-05 2020-03-09T00:25:23.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857087+0000", 2020-03-09T00:25:23.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7180000000000008e-06 2020-03-09T00:25:23.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857096+0000", 2020-03-09T00:25:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8880000000000003e-06 2020-03-09T00:25:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.857105+0000", 2020-03-09T00:25:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040913800000000004 2020-03-09T00:25:23.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.861197+0000", 2020-03-09T00:25:23.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3682e-05 2020-03-09T00:25:23.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.861210+0000", 2020-03-09T00:25:23.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002965152 2020-03-09T00:25:23.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864175+0000", 2020-03-09T00:25:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4409000000000002e-05 2020-03-09T00:25:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864210+0000", 2020-03-09T00:25:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072822809999999998 2020-03-09T00:25:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.768 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:23.769 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:23.769 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.894304527999999, 2020-03-09T00:25:23.769 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649, 2020-03-09T00:25:23.769 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.769 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.769 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.770 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.770 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:23.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:23.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022148699999999999 2020-03-09T00:25:23.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853554+0000", 2020-03-09T00:25:23.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7487e-05 2020-03-09T00:25:23.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853571+0000", 2020-03-09T00:25:23.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1065e-05 2020-03-09T00:25:23.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853582+0000", 2020-03-09T00:25:23.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7259999999999999e-06 2020-03-09T00:25:23.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853588+0000", 2020-03-09T00:25:23.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6578e-05 2020-03-09T00:25:23.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853634+0000", 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1282e-05 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853646+0000", 2020-03-09T00:25:23.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012718002000000001 2020-03-09T00:25:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866364+0000", 2020-03-09T00:25:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6638999999999999e-05 2020-03-09T00:25:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866400+0000", 2020-03-09T00:25:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649 2020-03-09T00:25:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:375 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.944856514000001, 2020-03-09T00:25:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317, 2020-03-09T00:25:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.778 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.779 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.779 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.779 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.779 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 375 2020-03-09T00:25:23.779 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.779 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:23.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.782 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:23.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001381363 2020-03-09T00:25:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802168+0000", 2020-03-09T00:25:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5639000000000001e-05 2020-03-09T00:25:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802184+0000", 2020-03-09T00:25:23.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6752000000000001e-05 2020-03-09T00:25:23.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802240+0000", 2020-03-09T00:25:23.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7064999999999999e-05 2020-03-09T00:25:23.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802268+0000", 2020-03-09T00:25:23.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6162999999999996e-05 2020-03-09T00:25:23.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802344+0000", 2020-03-09T00:25:23.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000166458 2020-03-09T00:25:23.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.787 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.788 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802510+0000", 2020-03-09T00:25:23.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018314410000000001 2020-03-09T00:25:23.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804342+0000", 2020-03-09T00:25:23.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079462270000000002 2020-03-09T00:25:23.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812288+0000", 2020-03-09T00:25:23.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078060580000000003 2020-03-09T00:25:23.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820094+0000", 2020-03-09T00:25:23.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110151 2020-03-09T00:25:23.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820204+0000", 2020-03-09T00:25:23.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317 2020-03-09T00:25:23.793 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.793 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.793 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:376 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.941501178999999, 2020-03-09T00:25:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127, 2020-03-09T00:25:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 376 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:23.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00133916 2020-03-09T00:25:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805481+0000", 2020-03-09T00:25:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1296e-05 2020-03-09T00:25:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805492+0000", 2020-03-09T00:25:23.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1435999999999999e-05 2020-03-09T00:25:23.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805514+0000", 2020-03-09T00:25:23.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5969999999999998e-06 2020-03-09T00:25:23.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805524+0000", 2020-03-09T00:25:23.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026245270000000002 2020-03-09T00:25:23.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808148+0000", 2020-03-09T00:25:23.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000157388 2020-03-09T00:25:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808305+0000", 2020-03-09T00:25:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00107099 2020-03-09T00:25:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.809376+0000", 2020-03-09T00:25:23.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071133840000000004 2020-03-09T00:25:23.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816490+0000", 2020-03-09T00:25:23.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038222479999999999 2020-03-09T00:25:23.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820312+0000", 2020-03-09T00:25:23.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012910100000000001 2020-03-09T00:25:23.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820441+0000", 2020-03-09T00:25:23.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:377 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~245760] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:23.805 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.937641766999999, 2020-03-09T00:25:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858, 2020-03-09T00:25:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.806 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 377 2020-03-09T00:25:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00047463399999999998 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808476+0000", 2020-03-09T00:25:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7579999999999998e-06 2020-03-09T00:25:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808482+0000", 2020-03-09T00:25:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7729999999999994e-06 2020-03-09T00:25:23.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808492+0000", 2020-03-09T00:25:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808496+0000", 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003930813 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812427+0000", 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013734399999999999 2020-03-09T00:25:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812564+0000", 2020-03-09T00:25:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00094040499999999995 2020-03-09T00:25:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.813505+0000", 2020-03-09T00:25:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030313029999999999 2020-03-09T00:25:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816536+0000", 2020-03-09T00:25:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048022810000000003 2020-03-09T00:25:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821338+0000", 2020-03-09T00:25:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4952999999999997e-05 2020-03-09T00:25:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821433+0000", 2020-03-09T00:25:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858 2020-03-09T00:25:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:378 3.2s0 3:482d6e06:::smithi10428818-17:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.937113643, 2020-03-09T00:25:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001, 2020-03-09T00:25:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 378 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2327999999999999e-05 2020-03-09T00:25:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808572+0000", 2020-03-09T00:25:23.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7400000000000002e-06 2020-03-09T00:25:23.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808576+0000", 2020-03-09T00:25:23.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5849999999999996e-06 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808582+0000", 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7070000000000001e-06 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808585+0000", 2020-03-09T00:25:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068015719999999997 2020-03-09T00:25:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815386+0000", 2020-03-09T00:25:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124704 2020-03-09T00:25:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815511+0000", 2020-03-09T00:25:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00063270399999999999 2020-03-09T00:25:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816144+0000", 2020-03-09T00:25:23.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056543100000000003 2020-03-09T00:25:23.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816709+0000", 2020-03-09T00:25:23.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048141440000000002 2020-03-09T00:25:23.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821523+0000", 2020-03-09T00:25:23.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012666000000000001 2020-03-09T00:25:23.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821650+0000", 2020-03-09T00:25:23.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:379 3.2s0 3:482d6e06:::smithi10428818-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:23.827 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.937033623000001, 2020-03-09T00:25:23.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999, 2020-03-09T00:25:23.828 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.828 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:23.828 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.828 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.828 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.828 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 379 2020-03-09T00:25:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:23.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3624000000000002e-05 2020-03-09T00:25:23.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808643+0000", 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.625e-06 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808647+0000", 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1500000000000002e-06 2020-03-09T00:25:23.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808654+0000", 2020-03-09T00:25:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8480000000000001e-06 2020-03-09T00:25:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808657+0000", 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079120670000000001 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816569+0000", 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012729000000000001 2020-03-09T00:25:23.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:23.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816696+0000", 2020-03-09T00:25:23.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054928110000000002 2020-03-09T00:25:23.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822189+0000", 2020-03-09T00:25:23.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6662000000000005e-05 2020-03-09T00:25:23.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822276+0000", 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001870029 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.824146+0000", 2020-03-09T00:25:23.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999 2020-03-09T00:25:23.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.837 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.837 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s2 55/54 ECSubWrite(tid=62, reqid=client.4751.0:402, at_version=55'26, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.808002+0000", 2020-03-09T00:25:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.93764157, 2020-03-09T00:25:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067101260000000003, 2020-03-09T00:25:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808002+0000", 2020-03-09T00:25:23.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808002+0000", 2020-03-09T00:25:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022239199999999999 2020-03-09T00:25:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808224+0000", 2020-03-09T00:25:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7348999999999999e-05 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808251+0000", 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1313000000000001e-05 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808273+0000", 2020-03-09T00:25:23.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6549999999999993e-06 2020-03-09T00:25:23.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808281+0000", 2020-03-09T00:25:23.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3673000000000001e-05 2020-03-09T00:25:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808315+0000", 2020-03-09T00:25:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2488e-05 2020-03-09T00:25:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.808328+0000", 2020-03-09T00:25:23.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063325869999999998 2020-03-09T00:25:23.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.814660+0000", 2020-03-09T00:25:23.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1669000000000003e-05 2020-03-09T00:25:23.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.814712+0000", 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067101260000000003 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.846 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s2 55/54 ECSubWrite(tid=67, reqid=client.4751.0:398, at_version=55'18, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.813412+0000", 2020-03-09T00:25:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.932231451, 2020-03-09T00:25:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066538960000000003, 2020-03-09T00:25:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813412+0000", 2020-03-09T00:25:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813412+0000", 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023050499999999999 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813642+0000", 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6463000000000001e-05 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813659+0000", 2020-03-09T00:25:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1501000000000001e-05 2020-03-09T00:25:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813670+0000", 2020-03-09T00:25:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5430000000000004e-06 2020-03-09T00:25:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813675+0000", 2020-03-09T00:25:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4376999999999999e-05 2020-03-09T00:25:23.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813719+0000", 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2407e-05 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.813732+0000", 2020-03-09T00:25:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006282953 2020-03-09T00:25:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.820014+0000", 2020-03-09T00:25:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1146999999999997e-05 2020-03-09T00:25:23.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.820066+0000", 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066538960000000003 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:503 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:23.855 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.833961+0000", 2020-03-09T00:25:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.911682596, 2020-03-09T00:25:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093609360000000003, 2020-03-09T00:25:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.856 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 503 2020-03-09T00:25:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.833961+0000", 2020-03-09T00:25:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.833961+0000", 2020-03-09T00:25:23.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9160999999999999e-05 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834010+0000", 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0647e-05 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834020+0000", 2020-03-09T00:25:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7700999999999999e-05 2020-03-09T00:25:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834038+0000", 2020-03-09T00:25:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1950000000000003e-06 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834046+0000", 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1390999999999997e-05 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834088+0000", 2020-03-09T00:25:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8152000000000006e-05 2020-03-09T00:25:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834176+0000", 2020-03-09T00:25:23.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064286300000000003 2020-03-09T00:25:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834819+0000", 2020-03-09T00:25:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038976039999999998 2020-03-09T00:25:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838716+0000", 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004520647 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843237+0000", 2020-03-09T00:25:23.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4574999999999996e-05 2020-03-09T00:25:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843322+0000", 2020-03-09T00:25:23.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093609360000000003 2020-03-09T00:25:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.866 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:504 3.2s0 3:482d6e06:::smithi10428818-17:head [rollback 15] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.834092+0000", 2020-03-09T00:25:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.911551057, 2020-03-09T00:25:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094498250000000002, 2020-03-09T00:25:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.867 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 504 2020-03-09T00:25:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834092+0000", 2020-03-09T00:25:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834092+0000", 2020-03-09T00:25:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6925999999999998e-05 2020-03-09T00:25:23.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834139+0000", 2020-03-09T00:25:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6800000000000004e-06 2020-03-09T00:25:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834146+0000", 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2583e-05 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834158+0000", 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3390000000000003e-06 2020-03-09T00:25:23.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834163+0000", 2020-03-09T00:25:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00100004 2020-03-09T00:25:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.835163+0000", 2020-03-09T00:25:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011991 2020-03-09T00:25:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.835283+0000", 2020-03-09T00:25:23.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092352899999999997 2020-03-09T00:25:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.836206+0000", 2020-03-09T00:25:23.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025829730000000001 2020-03-09T00:25:23.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838789+0000", 2020-03-09T00:25:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046750439999999997 2020-03-09T00:25:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843464+0000", 2020-03-09T00:25:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7800999999999996e-05 2020-03-09T00:25:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843542+0000", 2020-03-09T00:25:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094498250000000002 2020-03-09T00:25:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:505 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.9114342, 2020-03-09T00:25:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016627194000000001, 2020-03-09T00:25:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.879 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.880 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.880 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.880 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 505 2020-03-09T00:25:23.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.880 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:23.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:23.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7997000000000002e-05 2020-03-09T00:25:23.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834247+0000", 2020-03-09T00:25:23.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000001e-06 2020-03-09T00:25:23.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834251+0000", 2020-03-09T00:25:23.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.469e-06 2020-03-09T00:25:23.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834258+0000", 2020-03-09T00:25:23.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2660000000000002e-06 2020-03-09T00:25:23.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834262+0000", 2020-03-09T00:25:23.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032379409999999998 2020-03-09T00:25:23.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837500+0000", 2020-03-09T00:25:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013429800000000001 2020-03-09T00:25:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837634+0000", 2020-03-09T00:25:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076062099999999995 2020-03-09T00:25:23.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838394+0000", 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039555900000000003 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838790+0000", 2020-03-09T00:25:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01196025 2020-03-09T00:25:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.850750+0000", 2020-03-09T00:25:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5890999999999995e-05 2020-03-09T00:25:23.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.850836+0000", 2020-03-09T00:25:23.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016627194000000001 2020-03-09T00:25:23.889 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.889 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.890 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.890 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as3 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:23.890 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.914124+0000", 2020-03-09T00:25:23.890 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.831519059, 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094380360000000003, 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914124+0000", 2020-03-09T00:25:23.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914124+0000", 2020-03-09T00:25:23.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010665379999999999 2020-03-09T00:25:23.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915191+0000", 2020-03-09T00:25:23.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9760000000000001e-05 2020-03-09T00:25:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915210+0000", 2020-03-09T00:25:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2631e-05 2020-03-09T00:25:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915223+0000", 2020-03-09T00:25:23.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.143e-06 2020-03-09T00:25:23.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915227+0000", 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915287+0000", 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8060000000000006e-06 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915297+0000", 2020-03-09T00:25:23.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082301200000000005 2020-03-09T00:25:23.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923527+0000", 2020-03-09T00:25:23.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5521000000000001e-05 2020-03-09T00:25:23.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923562+0000", 2020-03-09T00:25:23.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094380360000000003 2020-03-09T00:25:23.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:587 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~589824] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.7694923239999998, 2020-03-09T00:25:23.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015855035, 2020-03-09T00:25:23.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.900 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.900 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.901 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 587 2020-03-09T00:25:23.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.901 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001387634 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977539+0000", 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3190000000000001e-06 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977548+0000", 2020-03-09T00:25:23.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5709000000000001e-05 2020-03-09T00:25:23.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977574+0000", 2020-03-09T00:25:23.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.471e-05 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977588+0000", 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8529000000000006e-05 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977677+0000", 2020-03-09T00:25:23.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013839599999999999 2020-03-09T00:25:23.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977815+0000", 2020-03-09T00:25:23.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015356079999999999 2020-03-09T00:25:23.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979351+0000", 2020-03-09T00:25:23.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070753639999999998 2020-03-09T00:25:23.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986426+0000", 2020-03-09T00:25:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054801509999999999 2020-03-09T00:25:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991906+0000", 2020-03-09T00:25:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9615000000000001e-05 2020-03-09T00:25:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992006+0000", 2020-03-09T00:25:23.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015855035 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:588 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~540672] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:23.910 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.7664346359999996, 2020-03-09T00:25:23.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013267434999999999, 2020-03-09T00:25:23.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.911 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.911 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.911 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.911 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.911 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 588 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:23.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001696822 2020-03-09T00:25:23.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980905+0000", 2020-03-09T00:25:23.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7710000000000006e-06 2020-03-09T00:25:23.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980913+0000", 2020-03-09T00:25:23.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4919999999999999e-05 2020-03-09T00:25:23.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980928+0000", 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.34e-06 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980935+0000", 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015720580000000001 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.982507+0000", 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7051000000000004e-05 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.982595+0000", 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081195499999999995 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983406+0000", 2020-03-09T00:25:23.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082914860000000007 2020-03-09T00:25:23.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991698+0000", 2020-03-09T00:25:23.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058976000000000005 2020-03-09T00:25:23.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992288+0000", 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018827199999999999 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992476+0000", 2020-03-09T00:25:23.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013267434999999999 2020-03-09T00:25:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:589 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~622592] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:23.920 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.7620877330000004, 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012556799, 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.921 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 589 2020-03-09T00:25:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:23.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019622569999999998 2020-03-09T00:25:23.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985518+0000", 2020-03-09T00:25:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3129999999999999e-06 2020-03-09T00:25:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985526+0000", 2020-03-09T00:25:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4384e-05 2020-03-09T00:25:23.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985540+0000", 2020-03-09T00:25:23.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2789999999999998e-06 2020-03-09T00:25:23.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985549+0000", 2020-03-09T00:25:23.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097411399999999999 2020-03-09T00:25:23.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986523+0000", 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5084000000000001e-05 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986598+0000", 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010282659999999999 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987626+0000", 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040731630000000003 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991699+0000", 2020-03-09T00:25:23.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043120320000000004 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996011+0000", 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100907 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996112+0000", 2020-03-09T00:25:23.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012556799 2020-03-09T00:25:23.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:590 3.2s0 3:49883c8e:::smithi10428818-40:head [setxattr _header (57) in=7b] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.7600343340000002, 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010789646, 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.931 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 590 2020-03-09T00:25:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2869999999999998e-05 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985692+0000", 2020-03-09T00:25:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.507e-06 2020-03-09T00:25:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985700+0000", 2020-03-09T00:25:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5087e-05 2020-03-09T00:25:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985715+0000", 2020-03-09T00:25:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5260000000000001e-06 2020-03-09T00:25:23.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985721+0000", 2020-03-09T00:25:23.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051320369999999999 2020-03-09T00:25:23.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.990853+0000", 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114726 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.990968+0000", 2020-03-09T00:25:23.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000440037 2020-03-09T00:25:23.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991408+0000", 2020-03-09T00:25:23.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001238942 2020-03-09T00:25:23.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992647+0000", 2020-03-09T00:25:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036263749999999998 2020-03-09T00:25:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996273+0000", 2020-03-09T00:25:23.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012553900000000001 2020-03-09T00:25:23.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996399+0000", 2020-03-09T00:25:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010789646 2020-03-09T00:25:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.941 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:591 3.2s0 3:49883c8e:::smithi10428818-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:23.942 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:23.942 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.7598751970000004, 2020-03-09T00:25:23.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015712046, 2020-03-09T00:25:23.942 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.942 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:23.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:23.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 591 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3307999999999999e-05 2020-03-09T00:25:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985821+0000", 2020-03-09T00:25:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5400000000000001e-06 2020-03-09T00:25:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985828+0000", 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2712e-05 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985841+0000", 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6920000000000001e-06 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985845+0000", 2020-03-09T00:25:23.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066716329999999997 2020-03-09T00:25:23.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992517+0000", 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012270799999999999 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992640+0000", 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00472662 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.997366+0000", 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012992200000000001 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.997496+0000", 2020-03-09T00:25:23.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039839109999999997 2020-03-09T00:25:23.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:18.001480+0000", 2020-03-09T00:25:23.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015712046 2020-03-09T00:25:23.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.950 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s6 68/67 ECSubWrite(tid=101, reqid=client.4751.0:592, at_version=68'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.989220+0000", 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.7564236590000002, 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007745259, 2020-03-09T00:25:23.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989220+0000", 2020-03-09T00:25:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:23.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989220+0000", 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050550200000000001 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:23.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989725+0000", 2020-03-09T00:25:23.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1642e-05 2020-03-09T00:25:23.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:23.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989747+0000", 2020-03-09T00:25:23.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6070999999999999e-05 2020-03-09T00:25:23.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989763+0000", 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0839999999999997e-06 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:23.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989769+0000", 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4284999999999999e-05 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989823+0000", 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4552000000000001e-05 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989838+0000", 2020-03-09T00:25:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070790940000000002 2020-03-09T00:25:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:23.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996917+0000", 2020-03-09T00:25:23.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8028999999999997e-05 2020-03-09T00:25:23.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:23.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:23.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:23.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996965+0000", 2020-03-09T00:25:23.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007745259 2020-03-09T00:25:23.958 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:23.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:23.959 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:23.960 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-03-09T00:25:23.962 INFO:tasks.rados.rados.0.smithi104.stdout:243: done (2 left) 2020-03-09T00:25:23.962 INFO:tasks.rados.rados.0.smithi104.stdout:244: done (1 left) 2020-03-09T00:25:23.962 INFO:tasks.rados.rados.0.smithi104.stdout:246: done (0 left) 2020-03-09T00:25:23.962 INFO:tasks.rados.rados.0.smithi104.stdout:247: snap_remove snap 20 2020-03-09T00:25:23.962 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:23.959+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:23.963 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:25:24.062 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:24.059+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:24.092 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/ceph_manager.py", line 124, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/ceph_manager.py", line 1210, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/ceph_manager.py", line 461, in kill_osd stderr=StringIO) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 145, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 683, in gevent._greenlet.Greenlet.get File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 303, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 274, in copy_to_log capture.write(line) TypeError: unbound method write() must be called with StringIO instance as first argument (got str instance instead) 2020-03-09T00:25:24.092 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/ceph_manager.py", line 1078, in do_thrash self._do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/ceph_manager.py", line 124, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/ceph_manager.py", line 1210, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/ceph_manager.py", line 461, in kill_osd stderr=StringIO) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 145, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 683, in gevent._greenlet.Greenlet.get File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 303, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 274, in copy_to_log capture.write(line) TypeError: unbound method write() must be called with StringIO instance as first argument (got str instance instead) 2020-03-09T00:25:24.093 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:24.093 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:24.094 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:24.094 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:24.103 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:25:24.163 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:24.162+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:24.234 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:24.235 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:24.235 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:24.235 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:24.235 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:24.243 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:25:24.264 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:24.262+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:24.363 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:24.364 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:24.364 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:24.364 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:24.364 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.364 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:242 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~638976] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:24.365 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:24.365 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 30.514644106999999, 2020-03-09T00:25:24.365 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001, 2020-03-09T00:25:24.365 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.365 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.365 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.365 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 242 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:24.366 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.367 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.367 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.367 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.367 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:24.367 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012420980000000001 2020-03-09T00:25:24.367 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.367 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.368 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.368 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847650+0000", 2020-03-09T00:25:24.368 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2689999999999996e-06 2020-03-09T00:25:24.368 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.368 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.368 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.369 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847657+0000", 2020-03-09T00:25:24.369 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2094e-05 2020-03-09T00:25:24.369 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.369 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.369 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.369 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847669+0000", 2020-03-09T00:25:24.370 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2690000000000002e-06 2020-03-09T00:25:24.370 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.370 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.370 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.370 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847676+0000", 2020-03-09T00:25:24.370 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3765999999999999e-05 2020-03-09T00:25:24.370 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.371 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.371 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.371 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847740+0000", 2020-03-09T00:25:24.371 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000155716 2020-03-09T00:25:24.371 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.371 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.371 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.372 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847896+0000", 2020-03-09T00:25:24.372 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020523669999999998 2020-03-09T00:25:24.372 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.372 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.372 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849948+0000", 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010549378999999999 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860497+0000", 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046388970000000003 2020-03-09T00:25:24.373 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.374 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.374 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.374 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865136+0000", 2020-03-09T00:25:24.374 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2514000000000005e-05 2020-03-09T00:25:24.374 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.374 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.374 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865199+0000", 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.375 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:243 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~376832] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:24.376 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:24.376 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 30.512138304, 2020-03-09T00:25:24.376 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999, 2020-03-09T00:25:24.376 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.376 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.377 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.377 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.377 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.378 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 243 2020-03-09T00:25:24.378 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.378 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.379 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.379 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.379 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:24.379 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.379 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.380 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.380 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.380 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:24.380 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012192539999999999 2020-03-09T00:25:24.380 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.380 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.381 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.381 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850133+0000", 2020-03-09T00:25:24.381 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1730000000000001e-06 2020-03-09T00:25:24.381 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.381 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.381 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.381 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850143+0000", 2020-03-09T00:25:24.382 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1009000000000001e-05 2020-03-09T00:25:24.382 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.382 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.382 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.382 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850174+0000", 2020-03-09T00:25:24.382 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0500000000000003e-06 2020-03-09T00:25:24.382 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.383 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.383 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.383 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850181+0000", 2020-03-09T00:25:24.383 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047873150000000003 2020-03-09T00:25:24.383 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.383 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.383 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.384 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.854968+0000", 2020-03-09T00:25:24.384 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013640000000000001 2020-03-09T00:25:24.384 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.384 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.384 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.384 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855104+0000", 2020-03-09T00:25:24.385 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001096669 2020-03-09T00:25:24.385 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.386 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.386 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856201+0000", 2020-03-09T00:25:24.386 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042977170000000004 2020-03-09T00:25:24.386 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.387 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.387 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.387 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860499+0000", 2020-03-09T00:25:24.387 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.008534827 2020-03-09T00:25:24.387 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.387 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.387 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869034+0000", 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4033e-05 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869118+0000", 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999 2020-03-09T00:25:24.388 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.389 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.389 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.389 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.389 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.389 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:244 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (56) in=7b] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:24.389 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:24.389 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 30.510793085, 2020-03-09T00:25:24.390 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999, 2020-03-09T00:25:24.390 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.390 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.390 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.390 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.390 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.391 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 244 2020-03-09T00:25:24.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000146107 2020-03-09T00:25:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850405+0000", 2020-03-09T00:25:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2099999999999999e-05 2020-03-09T00:25:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850418+0000", 2020-03-09T00:25:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9311000000000001e-05 2020-03-09T00:25:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850447+0000", 2020-03-09T00:25:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6460000000000004e-06 2020-03-09T00:25:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850453+0000", 2020-03-09T00:25:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011322660999999999 2020-03-09T00:25:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861776+0000", 2020-03-09T00:25:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015394900000000001 2020-03-09T00:25:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861930+0000", 2020-03-09T00:25:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076229599999999998 2020-03-09T00:25:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862692+0000", 2020-03-09T00:25:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002593092 2020-03-09T00:25:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865285+0000", 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039673670000000003 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869253+0000", 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010661 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.400 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869359+0000", 2020-03-09T00:25:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999 2020-03-09T00:25:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:245 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 30.510546850000001, 2020-03-09T00:25:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998, 2020-03-09T00:25:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 245 2020-03-09T00:25:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2926999999999997e-05 2020-03-09T00:25:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850548+0000", 2020-03-09T00:25:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7159999999999997e-06 2020-03-09T00:25:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850555+0000", 2020-03-09T00:25:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2537999999999999e-05 2020-03-09T00:25:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850568+0000", 2020-03-09T00:25:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6200000000000004e-06 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850573+0000", 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013277631 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863851+0000", 2020-03-09T00:25:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016975200000000001 2020-03-09T00:25:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864021+0000", 2020-03-09T00:25:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058362869999999999 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869857+0000", 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011197 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869969+0000", 2020-03-09T00:25:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0016289570000000001 2020-03-09T00:25:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.871598+0000", 2020-03-09T00:25:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998 2020-03-09T00:25:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:289 3.es0 3:73be088e:::smithi10428818-42:head [append 0~737280] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.546574614000001, 2020-03-09T00:25:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001, 2020-03-09T00:25:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289 2020-03-09T00:25:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001180787 2020-03-09T00:25:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815659+0000", 2020-03-09T00:25:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5960000000000004e-06 2020-03-09T00:25:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815667+0000", 2020-03-09T00:25:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7425e-05 2020-03-09T00:25:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.418 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815685+0000", 2020-03-09T00:25:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0162e-05 2020-03-09T00:25:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815695+0000", 2020-03-09T00:25:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2490000000000004e-05 2020-03-09T00:25:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815787+0000", 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7851999999999998e-05 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815885+0000", 2020-03-09T00:25:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001491497 2020-03-09T00:25:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817377+0000", 2020-03-09T00:25:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010162513999999999 2020-03-09T00:25:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827539+0000", 2020-03-09T00:25:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027651400000000001 2020-03-09T00:25:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.423 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830304+0000", 2020-03-09T00:25:24.423 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8808999999999995e-05 2020-03-09T00:25:24.423 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.423 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.423 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.423 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830403+0000", 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:292 3.es0 3:73be088e:::smithi10428818-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:24.424 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:24.425 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.543271591, 2020-03-09T00:25:24.425 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999, 2020-03-09T00:25:24.425 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.425 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:24.425 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.425 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.425 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.426 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 292 2020-03-09T00:25:24.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.426 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.426 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.426 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:24.426 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7518000000000001e-05 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.427 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817828+0000", 2020-03-09T00:25:24.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:24.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.428 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.428 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.428 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817834+0000", 2020-03-09T00:25:24.428 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2721000000000001e-05 2020-03-09T00:25:24.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.429 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.429 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817847+0000", 2020-03-09T00:25:24.429 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2339999999999996e-06 2020-03-09T00:25:24.429 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817851+0000", 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012178102999999999 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830029+0000", 2020-03-09T00:25:24.430 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012795100000000001 2020-03-09T00:25:24.431 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.431 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.431 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:24.431 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830157+0000", 2020-03-09T00:25:24.431 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003501414 2020-03-09T00:25:24.431 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.431 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833659+0000", 2020-03-09T00:25:24.432 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5081999999999996e-05 2020-03-09T00:25:24.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.432 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.432 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.432 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833744+0000", 2020-03-09T00:25:24.433 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002393077 2020-03-09T00:25:24.433 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.433 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.433 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.433 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.836137+0000", 2020-03-09T00:25:24.433 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999 2020-03-09T00:25:24.433 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:339 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.55336402, 2020-03-09T00:25:24.434 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998, 2020-03-09T00:25:24.435 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.435 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.435 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.435 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.435 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.435 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 339 2020-03-09T00:25:24.435 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.436 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.437 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:24.437 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00074630200000000001 2020-03-09T00:25:24.437 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.437 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.437 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.437 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808435+0000", 2020-03-09T00:25:24.437 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6087e-05 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808451+0000", 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.413e-05 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808465+0000", 2020-03-09T00:25:24.439 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4320000000000004e-06 2020-03-09T00:25:24.439 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.439 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.439 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.439 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808471+0000", 2020-03-09T00:25:24.440 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046408889999999996 2020-03-09T00:25:24.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.440 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.440 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.440 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813112+0000", 2020-03-09T00:25:24.440 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013929799999999999 2020-03-09T00:25:24.440 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.441 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.441 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813252+0000", 2020-03-09T00:25:24.441 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017257030000000001 2020-03-09T00:25:24.441 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.441 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.441 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.442 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.814977+0000", 2020-03-09T00:25:24.442 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099845400000000001 2020-03-09T00:25:24.442 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.442 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.442 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.442 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.824962+0000", 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049262399999999999 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825454+0000", 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.462e-05 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.443 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.444 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.444 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825539+0000", 2020-03-09T00:25:24.444 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998 2020-03-09T00:25:24.445 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.445 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.445 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.445 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.446 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.446 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:340 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~294912] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:24.446 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:24.446 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.550759591999999, 2020-03-09T00:25:24.446 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244, 2020-03-09T00:25:24.447 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.447 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.447 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.447 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.447 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.447 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 340 2020-03-09T00:25:24.447 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.448 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.448 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.448 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.448 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:24.448 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.448 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.449 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.449 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.449 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:24.449 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000216388 2020-03-09T00:25:24.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810509+0000", 2020-03-09T00:25:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6459999999999998e-06 2020-03-09T00:25:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810515+0000", 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.799e-06 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810524+0000", 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:24.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.452 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810528+0000", 2020-03-09T00:25:24.452 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007805228 2020-03-09T00:25:24.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.452 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818333+0000", 2020-03-09T00:25:24.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013953500000000001 2020-03-09T00:25:24.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.453 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.453 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.453 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818473+0000", 2020-03-09T00:25:24.453 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00094954999999999996 2020-03-09T00:25:24.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.454 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.454 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.819423+0000", 2020-03-09T00:25:24.454 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056089679999999998 2020-03-09T00:25:24.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825032+0000", 2020-03-09T00:25:24.455 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015242280000000001 2020-03-09T00:25:24.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.455 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.455 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826556+0000", 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4308000000000007e-05 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826640+0000", 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.456 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.457 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.457 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.457 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:341 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:24.457 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:24.457 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.550494509, 2020-03-09T00:25:24.457 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001, 2020-03-09T00:25:24.458 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.458 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.458 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.458 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.458 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.458 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 341 2020-03-09T00:25:24.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.459 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.459 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.459 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:24.459 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.460 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.460 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:24.460 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6538999999999997e-05 2020-03-09T00:25:24.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.460 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810594+0000", 2020-03-09T00:25:24.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3629999999999998e-06 2020-03-09T00:25:24.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.461 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.461 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810598+0000", 2020-03-09T00:25:24.461 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1715e-05 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810610+0000", 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7019999999999999e-06 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.462 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810613+0000", 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016583064000000002 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827196+0000", 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000134802 2020-03-09T00:25:24.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.464 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.464 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827331+0000", 2020-03-09T00:25:24.464 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079633999999999998 2020-03-09T00:25:24.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.465 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.465 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828127+0000", 2020-03-09T00:25:24.465 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000914331 2020-03-09T00:25:24.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.465 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.466 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.466 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.829042+0000", 2020-03-09T00:25:24.466 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045226199999999998 2020-03-09T00:25:24.466 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.466 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:24.467 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000101851 2020-03-09T00:25:24.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.467 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.467 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.467 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833666+0000", 2020-03-09T00:25:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001 2020-03-09T00:25:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:342 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.550416408, 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001, 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 342 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:24.470 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1574e-05 2020-03-09T00:25:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.472 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.472 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810658+0000", 2020-03-09T00:25:24.472 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.196e-06 2020-03-09T00:25:24.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.473 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.473 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810661+0000", 2020-03-09T00:25:24.473 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6000000000000003e-06 2020-03-09T00:25:24.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810667+0000", 2020-03-09T00:25:24.474 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3269999999999999e-06 2020-03-09T00:25:24.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.474 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.474 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810670+0000", 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018222015000000001 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828892+0000", 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107623 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:24.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828999+0000", 2020-03-09T00:25:24.476 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005589938 2020-03-09T00:25:24.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.476 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.476 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834589+0000", 2020-03-09T00:25:24.477 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012716299999999999 2020-03-09T00:25:24.477 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.478 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.478 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834716+0000", 2020-03-09T00:25:24.478 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048828580000000003 2020-03-09T00:25:24.478 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.478 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.478 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.478 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.839599+0000", 2020-03-09T00:25:24.479 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001 2020-03-09T00:25:24.479 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.479 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.479 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.479 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.479 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:374 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:24.480 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:24.480 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 20.566909585000001, 2020-03-09T00:25:24.480 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001, 2020-03-09T00:25:24.480 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.480 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:24.480 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.480 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.481 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.481 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 374 2020-03-09T00:25:24.481 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.481 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.481 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.481 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.481 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:24.482 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.482 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.482 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.482 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.482 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:24.482 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3215999999999998e-05 2020-03-09T00:25:24.482 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.483 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.483 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794216+0000", 2020-03-09T00:25:24.483 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7970000000000003e-06 2020-03-09T00:25:24.483 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.483 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.484 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794223+0000", 2020-03-09T00:25:24.484 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2177e-05 2020-03-09T00:25:24.484 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.484 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.484 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.485 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794235+0000", 2020-03-09T00:25:24.485 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8439999999999997e-06 2020-03-09T00:25:24.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.486 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.486 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794240+0000", 2020-03-09T00:25:24.486 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7449999999999997e-05 2020-03-09T00:25:24.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.487 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.487 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.487 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794337+0000", 2020-03-09T00:25:24.487 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0590000000000003e-06 2020-03-09T00:25:24.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.488 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:24.488 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794345+0000", 2020-03-09T00:25:24.488 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032359939999999998 2020-03-09T00:25:24.488 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.488 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.489 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797581+0000", 2020-03-09T00:25:24.489 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114491 2020-03-09T00:25:24.489 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.490 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.490 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.490 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797696+0000", 2020-03-09T00:25:24.490 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016180261000000001 2020-03-09T00:25:24.490 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.491 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.491 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.491 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.813876+0000", 2020-03-09T00:25:24.491 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001 2020-03-09T00:25:24.491 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.492 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.492 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.492 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.492 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.492 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:433 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~671744] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:24.492 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:24.492 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.559875658999999, 2020-03-09T00:25:24.493 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002, 2020-03-09T00:25:24.493 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.494 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.494 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.494 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.494 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.495 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 433 2020-03-09T00:25:24.495 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.495 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.495 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.495 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.495 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:24.495 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.496 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.496 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.496 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.496 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:24.497 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030767469999999999 2020-03-09T00:25:24.497 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.497 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804253+0000", 2020-03-09T00:25:24.498 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1990000000000006e-06 2020-03-09T00:25:24.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.498 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.498 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.498 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804262+0000", 2020-03-09T00:25:24.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9800999999999999e-05 2020-03-09T00:25:24.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.499 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.499 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.499 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804281+0000", 2020-03-09T00:25:24.499 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0343000000000001e-05 2020-03-09T00:25:24.499 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804292+0000", 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000220564 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.500 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804512+0000", 2020-03-09T00:25:24.501 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012449100000000001 2020-03-09T00:25:24.501 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.502 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.502 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804637+0000", 2020-03-09T00:25:24.502 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00145191 2020-03-09T00:25:24.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.503 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806089+0000", 2020-03-09T00:25:24.503 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020643070999999999 2020-03-09T00:25:24.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.503 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826732+0000", 2020-03-09T00:25:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081151190000000005 2020-03-09T00:25:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834847+0000", 2020-03-09T00:25:24.504 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1645999999999996e-05 2020-03-09T00:25:24.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.505 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.505 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834939+0000", 2020-03-09T00:25:24.505 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002 2020-03-09T00:25:24.505 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.505 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=66, reqid=client.4751.0:397, at_version=55'17, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.556688154, 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001, 2020-03-09T00:25:24.506 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.507 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:24.507 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.507 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.507 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.507 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:24.507 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104311 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.508 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804469+0000", 2020-03-09T00:25:24.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5208999999999999e-05 2020-03-09T00:25:24.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.509 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.509 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.509 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804494+0000", 2020-03-09T00:25:24.509 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5976000000000001e-05 2020-03-09T00:25:24.509 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.510 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804510+0000", 2020-03-09T00:25:24.510 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0229999999999998e-06 2020-03-09T00:25:24.510 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.510 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.510 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804516+0000", 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057666139999999998 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810282+0000", 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3182e-05 2020-03-09T00:25:24.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.512 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810296+0000", 2020-03-09T00:25:24.512 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016691906999999999 2020-03-09T00:25:24.512 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.512 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.512 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.513 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826987+0000", 2020-03-09T00:25:24.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8390000000000003e-05 2020-03-09T00:25:24.513 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.513 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.513 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.513 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827036+0000", 2020-03-09T00:25:24.513 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:434 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~770048] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:24.514 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.555946384999999, 2020-03-09T00:25:24.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003, 2020-03-09T00:25:24.515 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.515 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.515 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.515 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.515 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.515 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 434 2020-03-09T00:25:24.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.516 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.516 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.516 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:24.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.516 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:24.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011664830000000001 2020-03-09T00:25:24.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.518 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.518 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806272+0000", 2020-03-09T00:25:24.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9829999999999999e-06 2020-03-09T00:25:24.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.519 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806279+0000", 2020-03-09T00:25:24.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1960000000000001e-05 2020-03-09T00:25:24.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.519 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806291+0000", 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4999999999999999e-06 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806297+0000", 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075845660000000001 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.813882+0000", 2020-03-09T00:25:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013610299999999999 2020-03-09T00:25:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.814018+0000", 2020-03-09T00:25:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047231749999999996 2020-03-09T00:25:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.818741+0000", 2020-03-09T00:25:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01050245 2020-03-09T00:25:24.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829243+0000", 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0085708439999999993 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837814+0000", 2020-03-09T00:25:24.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4356000000000003e-05 2020-03-09T00:25:24.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.524 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.524 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837898+0000", 2020-03-09T00:25:24.524 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003 2020-03-09T00:25:24.524 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.525 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.525 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.525 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:435 3.es0 3:73e16f4f:::smithi10428818-13:head [setxattr _header (57) in=7b] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.554700611000001, 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003, 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.526 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 435 2020-03-09T00:25:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:24.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4781000000000001e-05 2020-03-09T00:25:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806427+0000", 2020-03-09T00:25:24.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9479999999999996e-06 2020-03-09T00:25:24.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.530 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806433+0000", 2020-03-09T00:25:24.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1838000000000001e-05 2020-03-09T00:25:24.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.530 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.530 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.530 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806444+0000", 2020-03-09T00:25:24.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7199999999999997e-06 2020-03-09T00:25:24.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806449+0000", 2020-03-09T00:25:24.531 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020656463999999999 2020-03-09T00:25:24.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827106+0000", 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013190700000000001 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827237+0000", 2020-03-09T00:25:24.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00078356200000000002 2020-03-09T00:25:24.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.533 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.533 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828021+0000", 2020-03-09T00:25:24.533 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001223678 2020-03-09T00:25:24.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829245+0000", 2020-03-09T00:25:24.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087850050000000002 2020-03-09T00:25:24.534 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.534 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.534 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838030+0000", 2020-03-09T00:25:24.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7771999999999998e-05 2020-03-09T00:25:24.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838127+0000", 2020-03-09T00:25:24.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003 2020-03-09T00:25:24.537 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.537 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.537 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.537 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:436 3.es0 3:73e16f4f:::smithi10428818-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-03-09T00:25:24.538 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:24.538 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.554564117000002, 2020-03-09T00:25:24.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998, 2020-03-09T00:25:24.538 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.538 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:24.538 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.538 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.539 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.539 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 436 2020-03-09T00:25:24.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.539 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3732000000000001e-05 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806532+0000", 2020-03-09T00:25:24.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5879999999999998e-06 2020-03-09T00:25:24.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.543 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806541+0000", 2020-03-09T00:25:24.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2738000000000001e-05 2020-03-09T00:25:24.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806553+0000", 2020-03-09T00:25:24.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.605e-06 2020-03-09T00:25:24.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806558+0000", 2020-03-09T00:25:24.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021728470999999999 2020-03-09T00:25:24.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828286+0000", 2020-03-09T00:25:24.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113791 2020-03-09T00:25:24.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:24.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828400+0000", 2020-03-09T00:25:24.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010563283 2020-03-09T00:25:24.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.549 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838963+0000", 2020-03-09T00:25:24.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121599 2020-03-09T00:25:24.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.550 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.839085+0000", 2020-03-09T00:25:24.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028128480000000002 2020-03-09T00:25:24.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.551 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.551 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.841898+0000", 2020-03-09T00:25:24.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998 2020-03-09T00:25:24.551 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.551 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.551 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.552 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=67, reqid=client.4751.0:398, at_version=55'18, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:24.552 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:24.552 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.550673343, 2020-03-09T00:25:24.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999, 2020-03-09T00:25:24.552 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.553 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:24.553 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:24.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:24.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000127076 2020-03-09T00:25:24.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810506+0000", 2020-03-09T00:25:24.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3247999999999998e-05 2020-03-09T00:25:24.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810529+0000", 2020-03-09T00:25:24.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4992e-05 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810544+0000", 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9669999999999998e-06 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810550+0000", 2020-03-09T00:25:24.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098081420000000006 2020-03-09T00:25:24.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820358+0000", 2020-03-09T00:25:24.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5410999999999999e-05 2020-03-09T00:25:24.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820374+0000", 2020-03-09T00:25:24.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066733160000000003 2020-03-09T00:25:24.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827047+0000", 2020-03-09T00:25:24.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4610999999999998e-05 2020-03-09T00:25:24.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.559 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.559 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827082+0000", 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:522 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:24.560 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:24.561 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.590302696, 2020-03-09T00:25:24.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018154887000000001, 2020-03-09T00:25:24.561 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.561 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.561 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.561 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 522 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:24.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:24.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7240000000000002e-05 2020-03-09T00:25:24.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770797+0000", 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1880000000000002e-06 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770805+0000", 2020-03-09T00:25:24.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5597e-05 2020-03-09T00:25:24.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770821+0000", 2020-03-09T00:25:24.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0109999999999997e-06 2020-03-09T00:25:24.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770827+0000", 2020-03-09T00:25:24.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00024047700000000001 2020-03-09T00:25:24.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771067+0000", 2020-03-09T00:25:24.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0305e-05 2020-03-09T00:25:24.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:24.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771078+0000", 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003975007 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.775053+0000", 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000152447 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.569 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.775205+0000", 2020-03-09T00:25:24.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010782299999999999 2020-03-09T00:25:24.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.569 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.569 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.776283+0000", 2020-03-09T00:25:24.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071855169999999998 2020-03-09T00:25:24.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783469+0000", 2020-03-09T00:25:24.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0053400169999999999 2020-03-09T00:25:24.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.788809+0000", 2020-03-09T00:25:24.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5851000000000001e-05 2020-03-09T00:25:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.788905+0000", 2020-03-09T00:25:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018154887000000001 2020-03-09T00:25:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.572 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.573 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.573 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.573 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:523 3.es0 3:73e16f4f:::smithi10428818-13:head [rollback 1a] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:24.573 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:24.573 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.59018092, 2020-03-09T00:25:24.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018315774999999999, 2020-03-09T00:25:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 523 2020-03-09T00:25:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.575 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:24.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:24.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3540000000000002e-05 2020-03-09T00:25:24.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770895+0000", 2020-03-09T00:25:24.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7960000000000002e-06 2020-03-09T00:25:24.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770899+0000", 2020-03-09T00:25:24.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.922e-06 2020-03-09T00:25:24.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770906+0000", 2020-03-09T00:25:24.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9869999999999999e-06 2020-03-09T00:25:24.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770909+0000", 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000235536 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771144+0000", 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4800000000000001e-06 2020-03-09T00:25:24.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:24.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771153+0000", 2020-03-09T00:25:24.580 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0091831280000000005 2020-03-09T00:25:24.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.780336+0000", 2020-03-09T00:25:24.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013816699999999999 2020-03-09T00:25:24.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.780474+0000", 2020-03-09T00:25:24.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00086282600000000002 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.781337+0000", 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002215828 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783553+0000", 2020-03-09T00:25:24.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0055424929999999999 2020-03-09T00:25:24.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.583 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.789095+0000", 2020-03-09T00:25:24.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2071999999999995e-05 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.789187+0000", 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018315774999999999 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.584 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.585 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:524 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:24.585 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:24.585 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.590113594, 2020-03-09T00:25:24.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020037222, 2020-03-09T00:25:24.585 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:24.586 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.586 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:24.586 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:24.586 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.586 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 524 2020-03-09T00:25:24.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.586 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:24.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:24.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:24.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:24.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:24.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:24.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2867999999999997e-05 2020-03-09T00:25:24.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:24.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770982+0000", 2020-03-09T00:25:24.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8850000000000004e-06 2020-03-09T00:25:24.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.589 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:24.589 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770988+0000", 2020-03-09T00:25:24.589 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1586000000000001e-05 2020-03-09T00:25:24.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770999+0000", 2020-03-09T00:25:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8590000000000004e-06 2020-03-09T00:25:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771004+0000", 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000171402 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771175+0000", 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2629999999999997e-06 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:24.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771183+0000", 2020-03-09T00:25:24.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011237143999999999 2020-03-09T00:25:24.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:24.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.782420+0000", 2020-03-09T00:25:24.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121636 2020-03-09T00:25:24.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:24.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.782541+0000", 2020-03-09T00:25:24.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00072058300000000001 2020-03-09T00:25:24.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783262+0000", 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044685599999999998 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:24.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783709+0000", 2020-03-09T00:25:24.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071821319999999999 2020-03-09T00:25:24.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:24.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.790891+0000", 2020-03-09T00:25:24.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5007999999999997e-05 2020-03-09T00:25:24.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:24.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:24.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:24.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.790976+0000", 2020-03-09T00:25:24.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020037222 2020-03-09T00:25:24.596 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.596 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.596 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.597 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:24.597 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:24.597 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:24.599 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:24.597+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:24.599 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:25:24.699 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:24.698+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:24.727 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:24.727 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:24.727 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:24.727 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:24.736 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:25:24.779 INFO:tasks.rados.rados.0.smithi104.stdout:247: done (0 left) 2020-03-09T00:25:24.779 INFO:tasks.rados.rados.0.smithi104.stdout:248: copy_from oid 24 from oid 41 current snap is 21 2020-03-09T00:25:24.780 INFO:tasks.rados.rados.0.smithi104.stdout:249: delete oid 20 current snap is 21 2020-03-09T00:25:24.786 INFO:tasks.rados.rados.0.smithi104.stdout:249: done (1 left) 2020-03-09T00:25:24.786 INFO:tasks.rados.rados.0.smithi104.stdout:250: copy_from oid 36 from oid 48 current snap is 21 2020-03-09T00:25:24.786 INFO:tasks.rados.rados.0.smithi104.stdout:251: rollback oid 32 current snap is 21 2020-03-09T00:25:24.786 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 32 to 18 2020-03-09T00:25:24.786 INFO:tasks.rados.rados.0.smithi104.stdout:252: rollback oid 28 current snap is 21 2020-03-09T00:25:24.786 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 28 to 19 2020-03-09T00:25:24.787 INFO:tasks.rados.rados.0.smithi104.stdout:253: read oid 37 snap -1 2020-03-09T00:25:24.787 INFO:tasks.rados.rados.0.smithi104.stdout:253: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:24.787 INFO:tasks.rados.rados.0.smithi104.stdout:254: rmattr oid 6 current snap is 21 2020-03-09T00:25:24.787 INFO:tasks.rados.rados.0.smithi104.stdout:254: done (5 left) 2020-03-09T00:25:24.787 INFO:tasks.rados.rados.0.smithi104.stdout:255: delete oid 1 current snap is 21 2020-03-09T00:25:24.787 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is unset 2020-03-09T00:25:24.797 INFO:tasks.rados.rados.0.smithi104.stdout:255: done (5 left) 2020-03-09T00:25:24.797 INFO:tasks.rados.rados.0.smithi104.stdout:256: snap_remove snap 17 2020-03-09T00:25:24.800 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:24.798+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:24.867 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:24.867 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:24.868 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:24.875 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:25:24.901 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:24.899+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:24.992 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:24.992 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:24.993 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:24.993 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:24.993 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.993 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:299 3.fs0 3:f75fa7c5:::smithi10428818-15:head [copy-from ver 4] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:24.993 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:24.993 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.171268478999998, 2020-03-09T00:25:24.994 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020952168, 2020-03-09T00:25:24.994 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:24.994 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:24.994 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:24.994 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:24.994 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:24.994 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 299 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:24.995 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:24.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:24.996 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012868800000000001 2020-03-09T00:25:24.996 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:24.996 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.996 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:24.996 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818244+0000", 2020-03-09T00:25:24.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5429999999999994e-06 2020-03-09T00:25:24.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:24.997 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.997 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:24.997 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818252+0000", 2020-03-09T00:25:24.997 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1410999999999999e-05 2020-03-09T00:25:24.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:24.998 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.998 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:24.998 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818314+0000", 2020-03-09T00:25:24.998 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9577000000000002e-05 2020-03-09T00:25:24.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818343+0000", 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8003e-05 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:24.999 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818401+0000", 2020-03-09T00:25:25.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014568900000000001 2020-03-09T00:25:25.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.000 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.000 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.000 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818547+0000", 2020-03-09T00:25:25.000 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0078751659999999994 2020-03-09T00:25:25.000 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.001 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.826422+0000", 2020-03-09T00:25:25.001 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023378689999999998 2020-03-09T00:25:25.001 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.001 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.001 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828760+0000", 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010241317999999999 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839001+0000", 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5903999999999998e-05 2020-03-09T00:25:25.002 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.003 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.003 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.003 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839067+0000", 2020-03-09T00:25:25.003 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020952168 2020-03-09T00:25:25.003 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.003 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.003 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.004 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.004 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.004 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:300 3.fs0 3:f75fa7c5:::smithi10428818-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:25.004 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:25.004 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.170949760999999, 2020-03-09T00:25:25.004 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021129146000000001, 2020-03-09T00:25:25.004 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 300 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.005 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:25.006 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.006 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.006 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.006 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.006 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:25.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2379000000000005e-05 2020-03-09T00:25:25.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.007 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.007 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.007 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818516+0000", 2020-03-09T00:25:25.007 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8039999999999998e-06 2020-03-09T00:25:25.007 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818522+0000", 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.208e-05 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.008 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818534+0000", 2020-03-09T00:25:25.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5580000000000002e-06 2020-03-09T00:25:25.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.009 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.009 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.009 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818539+0000", 2020-03-09T00:25:25.009 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060248300000000001 2020-03-09T00:25:25.009 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.819141+0000", 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1888e-05 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:25.010 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.819213+0000", 2020-03-09T00:25:25.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091353879999999995 2020-03-09T00:25:25.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.011 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.011 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.011 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828348+0000", 2020-03-09T00:25:25.011 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012281999999999999 2020-03-09T00:25:25.011 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:25.012 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828471+0000", 2020-03-09T00:25:25.012 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010931410000000001 2020-03-09T00:25:25.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.012 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.012 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839403+0000", 2020-03-09T00:25:25.013 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3260000000000001e-05 2020-03-09T00:25:25.013 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.013 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.013 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.013 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839466+0000", 2020-03-09T00:25:25.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7076000000000004e-05 2020-03-09T00:25:25.014 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.014 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.014 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.014 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839563+0000", 2020-03-09T00:25:25.014 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021129146000000001 2020-03-09T00:25:25.014 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.015 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.015 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.015 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.015 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:301 3.cs0 3:31ded997:::smithi10428818-32:head [append 0~753664] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:25.015 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.828231+0000", 2020-03-09T00:25:25.015 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.161152292000001, 2020-03-09T00:25:25.016 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017638918999999999, 2020-03-09T00:25:25.016 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.016 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.016 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.017 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 301 2020-03-09T00:25:25.017 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.017 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.017 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.017 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.017 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828231+0000", 2020-03-09T00:25:25.017 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.018 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.018 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828231+0000", 2020-03-09T00:25:25.018 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00059794500000000001 2020-03-09T00:25:25.018 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.018 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828829+0000", 2020-03-09T00:25:25.019 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4290000000000007e-06 2020-03-09T00:25:25.019 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.019 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.019 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.019 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828839+0000", 2020-03-09T00:25:25.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4939e-05 2020-03-09T00:25:25.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.020 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.020 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.020 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828864+0000", 2020-03-09T00:25:25.020 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3840000000000002e-06 2020-03-09T00:25:25.020 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.021 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.021 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828870+0000", 2020-03-09T00:25:25.021 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9761e-05 2020-03-09T00:25:25.021 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.021 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.022 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828920+0000", 2020-03-09T00:25:25.022 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012904799999999999 2020-03-09T00:25:25.022 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.022 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.022 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.023 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829049+0000", 2020-03-09T00:25:25.023 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015728630000000001 2020-03-09T00:25:25.023 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.023 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.023 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830622+0000", 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0071465979999999997 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837768+0000", 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080003439999999995 2020-03-09T00:25:25.024 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.025 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.025 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.845769+0000", 2020-03-09T00:25:25.025 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010160799999999999 2020-03-09T00:25:25.025 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.025 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.026 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.026 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.845870+0000", 2020-03-09T00:25:25.026 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017638918999999999 2020-03-09T00:25:25.026 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.026 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.026 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.026 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.027 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.027 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:304 3.cs0 3:31ded997:::smithi10428818-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:25.027 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:25.027 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.159418279, 2020-03-09T00:25:25.027 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018218758000000002, 2020-03-09T00:25:25.027 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.027 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.029 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.029 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.029 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.029 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 304 2020-03-09T00:25:25.029 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.029 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.030 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.030 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:25.030 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.030 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.030 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.031 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:25.031 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7318999999999999e-05 2020-03-09T00:25:25.031 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.031 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.031 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830013+0000", 2020-03-09T00:25:25.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1289999999999998e-06 2020-03-09T00:25:25.032 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.032 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.032 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.032 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830020+0000", 2020-03-09T00:25:25.032 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2313000000000001e-05 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830032+0000", 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7829999999999998e-06 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.033 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.034 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830037+0000", 2020-03-09T00:25:25.034 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007629449 2020-03-09T00:25:25.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.035 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.035 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.035 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837666+0000", 2020-03-09T00:25:25.035 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1544000000000003e-05 2020-03-09T00:25:25.035 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:25.036 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837758+0000", 2020-03-09T00:25:25.036 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0089964079999999991 2020-03-09T00:25:25.036 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.036 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.036 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846754+0000", 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113248 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846867+0000", 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001316565 2020-03-09T00:25:25.037 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.038 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.038 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.038 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.848184+0000", 2020-03-09T00:25:25.038 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018218758000000002 2020-03-09T00:25:25.038 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.038 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.038 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.039 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.039 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.039 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:344 3.fs0 3:f2571ca0:::smithi10428818-48:head [copy-from ver 35] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:25.039 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:25.039 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.177572369, 2020-03-09T00:25:25.039 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066389322000000001, 2020-03-09T00:25:25.039 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 344 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.040 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:25.041 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.041 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.041 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.041 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.041 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:25.042 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100208 2020-03-09T00:25:25.042 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.042 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.042 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.044 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811911+0000", 2020-03-09T00:25:25.044 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1372999999999999e-05 2020-03-09T00:25:25.044 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.045 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.045 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.045 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811923+0000", 2020-03-09T00:25:25.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5517999999999999e-05 2020-03-09T00:25:25.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.046 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.046 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.046 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811938+0000", 2020-03-09T00:25:25.046 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7190000000000004e-06 2020-03-09T00:25:25.046 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.047 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811945+0000", 2020-03-09T00:25:25.047 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1340000000000006e-05 2020-03-09T00:25:25.047 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.047 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.047 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812006+0000", 2020-03-09T00:25:25.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014919999999999999 2020-03-09T00:25:25.048 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.048 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.048 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.048 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812156+0000", 2020-03-09T00:25:25.048 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.038728380999999999 2020-03-09T00:25:25.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.049 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.049 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.049 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.850884+0000", 2020-03-09T00:25:25.049 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013700324 2020-03-09T00:25:25.049 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.050 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864584+0000", 2020-03-09T00:25:25.050 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013548961 2020-03-09T00:25:25.050 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.050 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.050 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878133+0000", 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7298000000000005e-05 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878200+0000", 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066389322000000001 2020-03-09T00:25:25.051 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.052 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.052 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.052 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.052 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.052 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:345 3.fs0 3:f2571ca0:::smithi10428818-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.177383787, 2020-03-09T00:25:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066989273000000002, 2020-03-09T00:25:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.053 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.054 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.054 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 345 2020-03-09T00:25:25.054 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.054 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.054 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:25.055 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5945000000000001e-05 2020-03-09T00:25:25.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.056 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.056 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812046+0000", 2020-03-09T00:25:25.056 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.545e-06 2020-03-09T00:25:25.056 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.056 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812052+0000", 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3108999999999999e-05 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812065+0000", 2020-03-09T00:25:25.057 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.566e-06 2020-03-09T00:25:25.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.058 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.058 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812070+0000", 2020-03-09T00:25:25.058 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00034552600000000003 2020-03-09T00:25:25.058 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.058 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812415+0000", 2020-03-09T00:25:25.059 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8084999999999997e-05 2020-03-09T00:25:25.059 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.059 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.059 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:25.059 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812474+0000", 2020-03-09T00:25:25.060 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.051585182 2020-03-09T00:25:25.060 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.061 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.061 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.061 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864059+0000", 2020-03-09T00:25:25.061 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8030999999999999e-05 2020-03-09T00:25:25.061 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:25.062 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864137+0000", 2020-03-09T00:25:25.062 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014720627 2020-03-09T00:25:25.062 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.062 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.062 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878857+0000", 2020-03-09T00:25:25.063 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9045999999999997e-05 2020-03-09T00:25:25.063 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.063 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.063 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.063 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878916+0000", 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2611e-05 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878989+0000", 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066989273000000002 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.064 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.065 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.065 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.065 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.065 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:380 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~557056] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:25.065 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:25.065 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.177543815, 2020-03-09T00:25:25.065 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019518607, 2020-03-09T00:25:25.066 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.066 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.066 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.066 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.066 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.066 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 380 2020-03-09T00:25:25.066 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.067 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:25.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036718500000000002 2020-03-09T00:25:25.068 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.068 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.068 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.068 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812207+0000", 2020-03-09T00:25:25.068 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3760000000000008e-06 2020-03-09T00:25:25.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.069 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.069 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812215+0000", 2020-03-09T00:25:25.069 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5866999999999999e-05 2020-03-09T00:25:25.069 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.069 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.070 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812231+0000", 2020-03-09T00:25:25.070 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2020-03-09T00:25:25.070 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.070 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.070 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812238+0000", 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6171000000000001e-05 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812304+0000", 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8213000000000003e-05 2020-03-09T00:25:25.071 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.072 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812402+0000", 2020-03-09T00:25:25.072 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012745790000000001 2020-03-09T00:25:25.072 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.072 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.072 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.813676+0000", 2020-03-09T00:25:25.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095836270000000008 2020-03-09T00:25:25.073 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.073 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.073 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.073 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.823260+0000", 2020-03-09T00:25:25.073 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080215159999999994 2020-03-09T00:25:25.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.074 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.074 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.074 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831282+0000", 2020-03-09T00:25:25.074 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6711999999999993e-05 2020-03-09T00:25:25.074 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.075 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.075 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.075 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831358+0000", 2020-03-09T00:25:25.075 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019518607 2020-03-09T00:25:25.075 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.075 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.075 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.076 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.076 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.076 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:384 3.fs0 3:fa16d489:::smithi10428818-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:25.076 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:25.076 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.171281773, 2020-03-09T00:25:25.076 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018476747000000002, 2020-03-09T00:25:25.076 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.077 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.077 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.077 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.077 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.077 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 384 2020-03-09T00:25:25.077 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.077 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.078 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.078 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:25.078 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.078 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.078 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1008999999999999e-05 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818153+0000", 2020-03-09T00:25:25.079 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2199999999999997e-06 2020-03-09T00:25:25.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.080 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.080 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818159+0000", 2020-03-09T00:25:25.080 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3512999999999999e-05 2020-03-09T00:25:25.080 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.080 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818173+0000", 2020-03-09T00:25:25.081 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5650000000000003e-06 2020-03-09T00:25:25.081 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.081 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.081 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.081 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818177+0000", 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01043207 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.828609+0000", 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216252 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.082 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:25.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.828825+0000", 2020-03-09T00:25:25.083 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005000539 2020-03-09T00:25:25.083 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.083 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.083 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.083 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833826+0000", 2020-03-09T00:25:25.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9849000000000001e-05 2020-03-09T00:25:25.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.084 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.084 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.084 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833906+0000", 2020-03-09T00:25:25.084 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026727299999999999 2020-03-09T00:25:25.084 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.085 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.085 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.085 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.836579+0000", 2020-03-09T00:25:25.085 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018476747000000002 2020-03-09T00:25:25.085 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.085 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.085 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:393 3.fs0 3:f1702e73:::smithi10428818-39:10 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 21.17082495, 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01890696, 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.086 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.087 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.087 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 393 2020-03-09T00:25:25.087 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.087 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.087 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:25.088 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.088 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.088 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.088 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.088 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1644000000000003e-05 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818600+0000", 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8680000000000001e-06 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.089 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818604+0000", 2020-03-09T00:25:25.090 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1230000000000002e-06 2020-03-09T00:25:25.090 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.090 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.090 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.090 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818611+0000", 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2399999999999999e-06 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818614+0000", 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015764628999999999 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.091 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.834379+0000", 2020-03-09T00:25:25.092 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016986400000000001 2020-03-09T00:25:25.092 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.092 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.092 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.092 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.834549+0000", 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002916592 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.837466+0000", 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01890696 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.093 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.094 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.094 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.094 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.094 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:544 3.fs0 3:f75fa7c5:::smithi10428818-15:head [append 0~491520] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:25.094 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:13.787862+0000", 2020-03-09T00:25:25.094 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.201521174, 2020-03-09T00:25:25.094 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022206677000000001, 2020-03-09T00:25:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 544 2020-03-09T00:25:25.095 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787862+0000", 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.096 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787862+0000", 2020-03-09T00:25:25.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027661299999999999 2020-03-09T00:25:25.097 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.097 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.097 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.097 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788139+0000", 2020-03-09T00:25:25.097 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9300000000000003e-06 2020-03-09T00:25:25.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.098 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.098 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788147+0000", 2020-03-09T00:25:25.098 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5483999999999999e-05 2020-03-09T00:25:25.098 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.098 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.100 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.100 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788162+0000", 2020-03-09T00:25:25.100 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4910000000000002e-06 2020-03-09T00:25:25.100 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.101 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.101 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.101 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788169+0000", 2020-03-09T00:25:25.101 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3137999999999998e-05 2020-03-09T00:25:25.101 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788222+0000", 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110845 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.102 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788333+0000", 2020-03-09T00:25:25.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012354709999999999 2020-03-09T00:25:25.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.103 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.103 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.103 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.789568+0000", 2020-03-09T00:25:25.103 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003034138 2020-03-09T00:25:25.103 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.104 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.792602+0000", 2020-03-09T00:25:25.104 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017400841 2020-03-09T00:25:25.104 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.104 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.104 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.105 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.810003+0000", 2020-03-09T00:25:25.105 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5726000000000001e-05 2020-03-09T00:25:25.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.105 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.105 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.106 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.810069+0000", 2020-03-09T00:25:25.106 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022206677000000001 2020-03-09T00:25:25.106 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.106 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.106 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.107 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.109 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.109 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:545 3.fs0 3:f75fa7c5:::smithi10428818-15:head [append 0~589824] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:25.109 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:13.792830+0000", 2020-03-09T00:25:25.109 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 11.196553757, 2020-03-09T00:25:25.109 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018188446, 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 545 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.110 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.111 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.111 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.792830+0000", 2020-03-09T00:25:25.111 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.111 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.111 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.113 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.113 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.792830+0000", 2020-03-09T00:25:25.113 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039823099999999998 2020-03-09T00:25:25.113 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.114 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.114 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.115 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.793228+0000", 2020-03-09T00:25:25.115 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9470000000000003e-06 2020-03-09T00:25:25.115 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.116 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.116 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.793236+0000", 2020-03-09T00:25:25.116 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3385e-05 2020-03-09T00:25:25.116 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.116 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.793249+0000", 2020-03-09T00:25:25.117 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5090000000000001e-06 2020-03-09T00:25:25.117 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.117 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.117 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.117 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.793255+0000", 2020-03-09T00:25:25.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3417000000000001e-05 2020-03-09T00:25:25.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.118 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.118 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.118 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.793298+0000", 2020-03-09T00:25:25.118 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012970000000000001 2020-03-09T00:25:25.118 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.119 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.793428+0000", 2020-03-09T00:25:25.119 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011483890000000001 2020-03-09T00:25:25.119 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.119 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.119 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.120 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.794576+0000", 2020-03-09T00:25:25.120 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072248130000000001 2020-03-09T00:25:25.120 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.120 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.120 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.801801+0000", 2020-03-09T00:25:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091686489999999992 2020-03-09T00:25:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.810970+0000", 2020-03-09T00:25:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8405999999999997e-05 2020-03-09T00:25:25.122 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.122 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.122 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.122 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.811018+0000", 2020-03-09T00:25:25.122 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018188446 2020-03-09T00:25:25.122 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.123 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.123 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.123 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.123 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.123 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:551 3.cs0 3:3776f87d:::smithi10428818-35:head [copy-from ver 24] snapc 1b=[1a,13,c] ondisk+write+known_if_redirected e65)", 2020-03-09T00:25:25.123 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:14.774998+0000", 2020-03-09T00:25:25.123 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.214385639, 2020-03-09T00:25:25.124 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.048260305000000003, 2020-03-09T00:25:25.124 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.124 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.124 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.124 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.124 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.124 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 551 2020-03-09T00:25:25.125 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.125 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.125 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.126 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.126 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.774998+0000", 2020-03-09T00:25:25.126 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.126 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.126 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.127 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.774998+0000", 2020-03-09T00:25:25.127 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3563999999999998e-05 2020-03-09T00:25:25.127 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.127 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.127 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775041+0000", 2020-03-09T00:25:25.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2859999999999996e-06 2020-03-09T00:25:25.128 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.128 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.128 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.128 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775048+0000", 2020-03-09T00:25:25.128 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1956e-05 2020-03-09T00:25:25.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.129 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.129 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775060+0000", 2020-03-09T00:25:25.129 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8690000000000004e-06 2020-03-09T00:25:25.129 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.129 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.130 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775068+0000", 2020-03-09T00:25:25.130 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0949000000000002e-05 2020-03-09T00:25:25.130 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.130 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.130 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775129+0000", 2020-03-09T00:25:25.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3609e-05 2020-03-09T00:25:25.131 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.131 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.131 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:25.131 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775142+0000", 2020-03-09T00:25:25.131 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023812970000000001 2020-03-09T00:25:25.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.132 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.132 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777523+0000", 2020-03-09T00:25:25.132 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5765000000000002e-05 2020-03-09T00:25:25.132 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.132 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777619+0000", 2020-03-09T00:25:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024822420000000001 2020-03-09T00:25:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.133 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.802442+0000", 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012621762999999999 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.815063+0000", 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081185110000000001 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.134 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.823182+0000", 2020-03-09T00:25:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6316000000000006e-05 2020-03-09T00:25:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.135 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.136 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.823258+0000", 2020-03-09T00:25:25.136 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.048260305000000003 2020-03-09T00:25:25.136 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.136 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.137 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.137 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.137 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.137 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:552 3.cs0 3:3776f87d:::smithi10428818-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e65)", 2020-03-09T00:25:25.137 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:14.775120+0000", 2020-03-09T00:25:25.137 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.214263299000001, 2020-03-09T00:25:25.138 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.049740894000000001, 2020-03-09T00:25:25.138 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.138 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.138 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.138 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.138 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.138 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 552 2020-03-09T00:25:25.139 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.139 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.139 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.139 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.139 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775120+0000", 2020-03-09T00:25:25.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.140 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.140 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.140 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.140 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775120+0000", 2020-03-09T00:25:25.140 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4431000000000002e-05 2020-03-09T00:25:25.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.141 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.141 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775165+0000", 2020-03-09T00:25:25.141 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7569999999999998e-06 2020-03-09T00:25:25.141 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.141 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775171+0000", 2020-03-09T00:25:25.142 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3426e-05 2020-03-09T00:25:25.142 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.142 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.142 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.142 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775185+0000", 2020-03-09T00:25:25.143 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2839999999999997e-06 2020-03-09T00:25:25.143 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.143 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.143 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.143 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775190+0000", 2020-03-09T00:25:25.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9645000000000002e-05 2020-03-09T00:25:25.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.144 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.144 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.144 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775250+0000", 2020-03-09T00:25:25.144 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0207000000000001e-05 2020-03-09T00:25:25.144 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:25.145 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775260+0000", 2020-03-09T00:25:25.145 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026141129999999999 2020-03-09T00:25:25.145 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.145 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.145 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.146 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777874+0000", 2020-03-09T00:25:25.146 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3913999999999999e-05 2020-03-09T00:25:25.146 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.147 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.147 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:25.147 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777928+0000", 2020-03-09T00:25:25.148 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.036313404000000001 2020-03-09T00:25:25.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.149 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.149 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.149 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.814241+0000", 2020-03-09T00:25:25.149 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6928999999999994e-05 2020-03-09T00:25:25.149 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.150 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:25.150 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.814328+0000", 2020-03-09T00:25:25.150 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010364329 2020-03-09T00:25:25.150 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.150 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.824693+0000", 2020-03-09T00:25:25.151 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4103000000000006e-05 2020-03-09T00:25:25.151 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.151 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.151 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.151 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.824757+0000", 2020-03-09T00:25:25.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010435200000000001 2020-03-09T00:25:25.152 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.152 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.152 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.152 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.824861+0000", 2020-03-09T00:25:25.152 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.049740894000000001 2020-03-09T00:25:25.152 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.153 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.153 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.153 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.153 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.153 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:572 3.cs0 3:3776f87d:::smithi10428818-35:head [copy-from ver 6] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:25.153 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.943798+0000", 2020-03-09T00:25:25.153 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.045585838, 2020-03-09T00:25:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021300082000000001, 2020-03-09T00:25:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 572 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943798+0000", 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.155 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943798+0000", 2020-03-09T00:25:25.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1941000000000003e-05 2020-03-09T00:25:25.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943850+0000", 2020-03-09T00:25:25.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7620000000000001e-06 2020-03-09T00:25:25.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943855+0000", 2020-03-09T00:25:25.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4925e-05 2020-03-09T00:25:25.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943870+0000", 2020-03-09T00:25:25.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9259999999999999e-06 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943875+0000", 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9487000000000003e-05 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943915+0000", 2020-03-09T00:25:25.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012597699999999999 2020-03-09T00:25:25.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.944041+0000", 2020-03-09T00:25:25.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003912436 2020-03-09T00:25:25.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.947953+0000", 2020-03-09T00:25:25.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033233059999999998 2020-03-09T00:25:25.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.951276+0000", 2020-03-09T00:25:25.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013528966999999999 2020-03-09T00:25:25.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.964805+0000", 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000292355 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965098+0000", 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021300082000000001 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.163 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.164 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:573 3.cs0 3:3776f87d:::smithi10428818-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:25.164 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.943913+0000", 2020-03-09T00:25:25.164 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.0454705190000002, 2020-03-09T00:25:25.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021618235, 2020-03-09T00:25:25.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.165 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.165 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.165 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 573 2020-03-09T00:25:25.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.166 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.166 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.166 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943913+0000", 2020-03-09T00:25:25.166 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.166 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943913+0000", 2020-03-09T00:25:25.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6050999999999998e-05 2020-03-09T00:25:25.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943939+0000", 2020-03-09T00:25:25.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.557e-06 2020-03-09T00:25:25.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943943+0000", 2020-03-09T00:25:25.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1754e-05 2020-03-09T00:25:25.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943954+0000", 2020-03-09T00:25:25.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2179999999999999e-06 2020-03-09T00:25:25.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943958+0000", 2020-03-09T00:25:25.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021570000000000001 2020-03-09T00:25:25.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.944173+0000", 2020-03-09T00:25:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1937000000000002e-05 2020-03-09T00:25:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:25.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.944215+0000", 2020-03-09T00:25:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069155550000000003 2020-03-09T00:25:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.951131+0000", 2020-03-09T00:25:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013795300000000001 2020-03-09T00:25:25.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.951269+0000", 2020-03-09T00:25:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013891163999999999 2020-03-09T00:25:25.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965160+0000", 2020-03-09T00:25:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011575500000000001 2020-03-09T00:25:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965276+0000", 2020-03-09T00:25:25.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000255591 2020-03-09T00:25:25.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965531+0000", 2020-03-09T00:25:25.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021618235 2020-03-09T00:25:25.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.176 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.176 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:634 3.cs0 3:32d16cfa:::smithi10428818-24:head [copy-from ver 4] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:25.176 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.778929+0000", 2020-03-09T00:25:25.177 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.21045464899999999, 2020-03-09T00:25:25.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027938032000000002, 2020-03-09T00:25:25.177 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.177 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.178 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.178 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.178 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.178 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 634 2020-03-09T00:25:25.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.178 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778929+0000", 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778929+0000", 2020-03-09T00:25:25.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0008e-05 2020-03-09T00:25:25.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778969+0000", 2020-03-09T00:25:25.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9050000000000002e-06 2020-03-09T00:25:25.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778975+0000", 2020-03-09T00:25:25.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0200000000000001e-05 2020-03-09T00:25:25.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778985+0000", 2020-03-09T00:25:25.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1460000000000001e-06 2020-03-09T00:25:25.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778991+0000", 2020-03-09T00:25:25.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0701000000000001e-05 2020-03-09T00:25:25.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779042+0000", 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6040000000000004e-06 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:25.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779049+0000", 2020-03-09T00:25:25.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043638970000000003 2020-03-09T00:25:25.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783413+0000", 2020-03-09T00:25:25.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013153199999999999 2020-03-09T00:25:25.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783545+0000", 2020-03-09T00:25:25.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016497833 2020-03-09T00:25:25.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.800043+0000", 2020-03-09T00:25:25.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055694259999999997 2020-03-09T00:25:25.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805612+0000", 2020-03-09T00:25:25.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010821209999999999 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806694+0000", 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000172659 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806867+0000", 2020-03-09T00:25:25.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027938032000000002 2020-03-09T00:25:25.188 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.188 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:635 3.cs0 3:32d16cfa:::smithi10428818-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e75)", 2020-03-09T00:25:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.779023+0000", 2020-03-09T00:25:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.210360767, 2020-03-09T00:25:25.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.028096716000000001, 2020-03-09T00:25:25.190 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.190 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:25.190 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.190 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.190 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.190 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 635 2020-03-09T00:25:25.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.191 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779023+0000", 2020-03-09T00:25:25.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779023+0000", 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4238000000000001e-05 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779047+0000", 2020-03-09T00:25:25.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4230000000000001e-06 2020-03-09T00:25:25.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.193 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779050+0000", 2020-03-09T00:25:25.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2509999999999997e-06 2020-03-09T00:25:25.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779058+0000", 2020-03-09T00:25:25.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.785e-06 2020-03-09T00:25:25.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779060+0000", 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0540999999999997e-05 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779101+0000", 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5169999999999999e-06 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:25.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779107+0000", 2020-03-09T00:25:25.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046296230000000002 2020-03-09T00:25:25.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783736+0000", 2020-03-09T00:25:25.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1681000000000002e-05 2020-03-09T00:25:25.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:25.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783788+0000", 2020-03-09T00:25:25.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020864237000000001 2020-03-09T00:25:25.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804652+0000", 2020-03-09T00:25:25.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110419 2020-03-09T00:25:25.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:25.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804762+0000", 2020-03-09T00:25:25.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021448970000000002 2020-03-09T00:25:25.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806907+0000", 2020-03-09T00:25:25.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3344999999999997e-05 2020-03-09T00:25:25.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806971+0000", 2020-03-09T00:25:25.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014875899999999999 2020-03-09T00:25:25.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.807119+0000", 2020-03-09T00:25:25.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.028096716000000001 2020-03-09T00:25:25.201 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:639 3.cs0 3:31ded997:::smithi10428818-32:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785921+0000", 2020-03-09T00:25:25.202 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.20346214300000001, 2020-03-09T00:25:25.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020119509000000001, 2020-03-09T00:25:25.203 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.203 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.203 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.203 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.204 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.204 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 639 2020-03-09T00:25:25.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.204 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785921+0000", 2020-03-09T00:25:25.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785921+0000", 2020-03-09T00:25:25.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4159000000000002e-05 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785966+0000", 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.844e-06 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785972+0000", 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785985+0000", 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1830000000000002e-06 2020-03-09T00:25:25.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785990+0000", 2020-03-09T00:25:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053932700000000004 2020-03-09T00:25:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786530+0000", 2020-03-09T00:25:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018090599999999999 2020-03-09T00:25:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786711+0000", 2020-03-09T00:25:25.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081753880000000004 2020-03-09T00:25:25.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.794886+0000", 2020-03-09T00:25:25.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010617161 2020-03-09T00:25:25.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805503+0000", 2020-03-09T00:25:25.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039746399999999999 2020-03-09T00:25:25.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805901+0000", 2020-03-09T00:25:25.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000140323 2020-03-09T00:25:25.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806041+0000", 2020-03-09T00:25:25.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020119509000000001 2020-03-09T00:25:25.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.213 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.213 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:640 3.cs0 3:31ded997:::smithi10428818-32:head [rollback 23] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.786041+0000", 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.20334289, 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020266050000000001, 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.214 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.215 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 640 2020-03-09T00:25:25.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.215 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786041+0000", 2020-03-09T00:25:25.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786041+0000", 2020-03-09T00:25:25.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6978999999999999e-05 2020-03-09T00:25:25.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786088+0000", 2020-03-09T00:25:25.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4779999999999996e-06 2020-03-09T00:25:25.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786094+0000", 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1922e-05 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786106+0000", 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6349999999999997e-06 2020-03-09T00:25:25.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786111+0000", 2020-03-09T00:25:25.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0094916340000000005 2020-03-09T00:25:25.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.795602+0000", 2020-03-09T00:25:25.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013348799999999999 2020-03-09T00:25:25.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.795736+0000", 2020-03-09T00:25:25.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082019099999999995 2020-03-09T00:25:25.221 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796556+0000", 2020-03-09T00:25:25.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0090484759999999997 2020-03-09T00:25:25.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805604+0000", 2020-03-09T00:25:25.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00054049499999999999 2020-03-09T00:25:25.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806145+0000", 2020-03-09T00:25:25.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016175200000000001 2020-03-09T00:25:25.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806307+0000", 2020-03-09T00:25:25.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020266050000000001 2020-03-09T00:25:25.224 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.224 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.224 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.225 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:641 3.cs0 3:31ded997:::smithi10428818-32:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:25.225 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.786157+0000", 2020-03-09T00:25:25.225 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.20322702100000001, 2020-03-09T00:25:25.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024528417, 2020-03-09T00:25:25.226 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:25.226 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:25.226 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:25.226 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:25.226 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:25.227 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 641 2020-03-09T00:25:25.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.227 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:25.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:25.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786157+0000", 2020-03-09T00:25:25.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:25.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:25.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786157+0000", 2020-03-09T00:25:25.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2144000000000002e-05 2020-03-09T00:25:25.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:25.229 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786199+0000", 2020-03-09T00:25:25.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1720000000000004e-06 2020-03-09T00:25:25.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.229 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:25.230 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786205+0000", 2020-03-09T00:25:25.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2237e-05 2020-03-09T00:25:25.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:25.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786217+0000", 2020-03-09T00:25:25.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5460000000000002e-06 2020-03-09T00:25:25.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786222+0000", 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018580409999999999 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804802+0000", 2020-03-09T00:25:25.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3192e-05 2020-03-09T00:25:25.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:25.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804875+0000", 2020-03-09T00:25:25.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046342300000000003 2020-03-09T00:25:25.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:25.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805339+0000", 2020-03-09T00:25:25.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039975600000000003 2020-03-09T00:25:25.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805738+0000", 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048028309999999996 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.810541+0000", 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014370600000000001 2020-03-09T00:25:25.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:25.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:25.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:25.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.810685+0000", 2020-03-09T00:25:25.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024528417 2020-03-09T00:25:25.236 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.236 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.236 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.237 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:25.237 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:25.237 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:25.237 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:25.239 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:25.237+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:25.241 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:25:25.351 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:25.349+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:25.372 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:25.372 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:25.372 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:25.372 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:25.377 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:25:25.381 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:25.402 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:25.423 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:25.443 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:25.451 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:25.449+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:25.462 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:25.484 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:25.493 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:25.494 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:25.494 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:25.494 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:25.494 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:25.502 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:25:25.504 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:25.524 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:25.544 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:25.552 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:25.550+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:25.564 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:25.584 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:25.604 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:25.618 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:25.618 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:25.623 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:25.626 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:25:25.644 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:25.644 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:25.642+0000 7fcad67fc700 0 -- 172.21.15.148:0/1720283754 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7fcabc02f8d0 msgr2=0x7fcabc031ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:25.653 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:25.651+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:25.753 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:25.751+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:25.763 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:25.763 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:25.772 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:25:25.785 INFO:tasks.rados.rados.0.smithi104.stdout:256: done (5 left) 2020-03-09T00:25:25.785 INFO:tasks.rados.rados.0.smithi104.stdout:257: read oid 8 snap -1 2020-03-09T00:25:25.786 INFO:tasks.rados.rados.0.smithi104.stdout:257: expect (ObjNum 74 snap 11 seq_num 74) 2020-03-09T00:25:25.790 INFO:tasks.rados.rados.0.smithi104.stdout:258: rmattr oid 13 current snap is 21 2020-03-09T00:25:25.791 INFO:tasks.rados.rados.0.smithi104.stdout:258: done (6 left) 2020-03-09T00:25:25.791 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 21 current snap is 21 2020-03-09T00:25:25.791 INFO:tasks.rados.rados.0.smithi104.stdout:259: seq_num 110 ranges {1769472=688128,2457600=507904,2965504=655360} 2020-03-09T00:25:25.800 INFO:tasks.rados.rados.0.smithi104.stdout:259: writing smithi10428818-21 from 1769472 to 2457600 tid 1 2020-03-09T00:25:25.803 INFO:tasks.rados.rados.0.smithi104.stdout:259: writing smithi10428818-21 from 2457600 to 2965504 tid 2 2020-03-09T00:25:25.807 INFO:tasks.rados.rados.0.smithi104.stdout:259: writing smithi10428818-21 from 2965504 to 3620864 tid 3 2020-03-09T00:25:25.807 INFO:tasks.rados.rados.0.smithi104.stdout:260: snap_remove snap 14 2020-03-09T00:25:25.854 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:25.853+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:25.865 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:25.866 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.866 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.867 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.868 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.868 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.868 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.868 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.868 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.869 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.869 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:25.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:25.937 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:25.937 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:25.946 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:25:25.955 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:25.953+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:26.056 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:26.054+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:26.072 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:26.072 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:26.081 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:25:26.157 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:26.155+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:26.198 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:26.198 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:26.206 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:25:26.258 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:26.255+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:26.326 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:26.326 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:26.330 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:25:26.371 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:26.369+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:26.459 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:26.458+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:26.486 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:26.486 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:26.495 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:25:26.560 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:26.559+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:26.621 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:26.622 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:26.630 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:25:26.661 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:26.659+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:26.745 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:26.746 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:26.754 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:25:26.761 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:26.760+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:26.788 INFO:tasks.rados.rados.0.smithi104.stdout:260: done (7 left) 2020-03-09T00:25:26.788 INFO:tasks.rados.rados.0.smithi104.stdout:261: read oid 40 snap 19 2020-03-09T00:25:26.788 INFO:tasks.rados.rados.0.smithi104.stdout:261: expect (ObjNum 104 snap 17 seq_num 104) 2020-03-09T00:25:26.789 INFO:tasks.rados.rados.0.smithi104.stdout:262: snap_create 2020-03-09T00:25:26.789 INFO:tasks.rados.rados.0.smithi104.stdout:252: finishing rollback tid 0 to smithi10428818-28 2020-03-09T00:25:26.789 INFO:tasks.rados.rados.0.smithi104.stdout:253: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:26.803 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-03-09T00:25:26.809 INFO:tasks.rados.rados.0.smithi104.stdout:252: finishing rollback tid 1 to smithi10428818-28 2020-03-09T00:25:26.809 INFO:tasks.rados.rados.0.smithi104.stdout:252: finishing rollback tid 2 to smithi10428818-28 2020-03-09T00:25:26.809 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 28 v 60 (ObjNum 89 snap 14 seq_num 89) dirty exists 2020-03-09T00:25:26.809 INFO:tasks.rados.rados.0.smithi104.stdout:251: finishing rollback tid 0 to smithi10428818-32 2020-03-09T00:25:26.809 INFO:tasks.rados.rados.0.smithi104.stdout:251: finishing rollback tid 1 to smithi10428818-32 2020-03-09T00:25:26.809 INFO:tasks.rados.rados.0.smithi104.stdout:248: finishing copy_from to smithi10428818-24 2020-03-09T00:25:26.810 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 24 v 4 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-03-09T00:25:26.810 INFO:tasks.rados.rados.0.smithi104.stdout:248: finishing copy_from racing read to smithi10428818-24 2020-03-09T00:25:26.810 INFO:tasks.rados.rados.0.smithi104.stdout:251: finishing rollback tid 2 to smithi10428818-32 2020-03-09T00:25:26.810 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 32 v 75 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-03-09T00:25:26.810 INFO:tasks.rados.rados.0.smithi104.stdout:250: finishing copy_from to smithi10428818-36 2020-03-09T00:25:26.810 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 36 v 35 (ObjNum 71 snap 11 seq_num 71) dirty exists 2020-03-09T00:25:26.811 INFO:tasks.rados.rados.0.smithi104.stdout:250: finishing copy_from racing read to smithi10428818-36 2020-03-09T00:25:26.811 INFO:tasks.rados.rados.0.smithi104.stdout:257: expect (ObjNum 74 snap 11 seq_num 74) 2020-03-09T00:25:26.862 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:26.860+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:26.879 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:26.879 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:26.892 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:25:26.941 INFO:tasks.rados.rados.0.smithi104.stdout:259: finishing write tid 1 to smithi10428818-21 2020-03-09T00:25:26.942 INFO:tasks.rados.rados.0.smithi104.stdout:259: finishing write tid 2 to smithi10428818-21 2020-03-09T00:25:26.942 INFO:tasks.rados.rados.0.smithi104.stdout:259: finishing write tid 3 to smithi10428818-21 2020-03-09T00:25:26.942 INFO:tasks.rados.rados.0.smithi104.stdout:259: finishing write tid 5 to smithi10428818-21 2020-03-09T00:25:26.942 INFO:tasks.rados.rados.0.smithi104.stdout:259: finishing write tid 6 to smithi10428818-21 2020-03-09T00:25:26.942 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 21 v 31 (ObjNum 110 snap 21 seq_num 110) dirty exists 2020-03-09T00:25:26.942 INFO:tasks.rados.rados.0.smithi104.stdout:259: left oid 21 (ObjNum 110 snap 21 seq_num 110) 2020-03-09T00:25:26.943 INFO:tasks.rados.rados.0.smithi104.stdout:261: expect (ObjNum 104 snap 17 seq_num 104) 2020-03-09T00:25:26.963 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:26.961+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:27.015 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:27.015 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:27.024 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:25:27.064 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:27.062+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:27.138 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:27.139 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:27.147 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:25:27.165 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:27.163+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:27.263 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:27.263 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:27.265 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:27.263+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:27.271 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:25:27.366 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:27.364+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:27.409 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:27.409 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:27.418 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:25:27.467 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:27.464+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:27.542 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:27.542 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:27.551 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:25:27.567 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:27.566+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:27.668 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:27.667+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:27.714 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:27.715 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:27.723 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:25:27.770 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:27.768+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:27.797 INFO:tasks.rados.rados.0.smithi104.stdout:248: done (8 left) 2020-03-09T00:25:27.798 INFO:tasks.rados.rados.0.smithi104.stdout:250: done (7 left) 2020-03-09T00:25:27.798 INFO:tasks.rados.rados.0.smithi104.stdout:251: done (6 left) 2020-03-09T00:25:27.798 INFO:tasks.rados.rados.0.smithi104.stdout:252: done (5 left) 2020-03-09T00:25:27.798 INFO:tasks.rados.rados.0.smithi104.stdout:253: done (4 left) 2020-03-09T00:25:27.798 INFO:tasks.rados.rados.0.smithi104.stdout:257: done (3 left) 2020-03-09T00:25:27.798 INFO:tasks.rados.rados.0.smithi104.stdout:259: done (2 left) 2020-03-09T00:25:27.798 INFO:tasks.rados.rados.0.smithi104.stdout:261: done (1 left) 2020-03-09T00:25:27.799 INFO:tasks.rados.rados.0.smithi104.stdout:262: done (0 left) 2020-03-09T00:25:27.799 INFO:tasks.rados.rados.0.smithi104.stdout:263: read oid 47 snap -1 2020-03-09T00:25:27.799 INFO:tasks.rados.rados.0.smithi104.stdout:263: expect (ObjNum 46 snap 0 seq_num 46) 2020-03-09T00:25:27.799 INFO:tasks.rados.rados.0.smithi104.stdout:264: snap_create 2020-03-09T00:25:27.809 INFO:tasks.rados.rados.0.smithi104.stdout:263: expect (ObjNum 46 snap 0 seq_num 46) 2020-03-09T00:25:27.809 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is set 2020-03-09T00:25:27.846 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:27.846 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:27.855 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:25:27.871 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:27.869+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:27.877 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:27.972 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:27.970+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:28.016 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:28.016 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:28.025 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:25:28.032 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:28.062 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:28.072 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:28.071+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:28.076 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:28.097 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:28.118 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:28.139 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:28.153 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:28.153 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:28.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:28.153 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:28.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.154 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:28.154 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:28.154 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.300625912999998, 2020-03-09T00:25:28.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999, 2020-03-09T00:25:28.154 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.154 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017622699999999999 2020-03-09T00:25:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850137+0000", 2020-03-09T00:25:28.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1974000000000003e-05 2020-03-09T00:25:28.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850169+0000", 2020-03-09T00:25:28.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4702000000000002e-05 2020-03-09T00:25:28.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850193+0000", 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5790000000000004e-06 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.158 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850202+0000", 2020-03-09T00:25:28.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013641599999999999 2020-03-09T00:25:28.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.159 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850338+0000", 2020-03-09T00:25:28.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4691999999999999e-05 2020-03-09T00:25:28.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850353+0000", 2020-03-09T00:25:28.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011849303 2020-03-09T00:25:28.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862202+0000", 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1013000000000001e-05 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862253+0000", 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999 2020-03-09T00:25:28.161 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.162 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.162 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.162 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:28.162 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:28.162 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.294430509000001, 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595, 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:28.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.164 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.164 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:28.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107948 2020-03-09T00:25:28.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856264+0000", 2020-03-09T00:25:28.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4422000000000001e-05 2020-03-09T00:25:28.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856288+0000", 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5265e-05 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856304+0000", 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3209999999999994e-06 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856313+0000", 2020-03-09T00:25:28.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00056587900000000003 2020-03-09T00:25:28.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856879+0000", 2020-03-09T00:25:28.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1976e-05 2020-03-09T00:25:28.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856891+0000", 2020-03-09T00:25:28.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098355160000000007 2020-03-09T00:25:28.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.170 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.170 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866726+0000", 2020-03-09T00:25:28.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6267999999999996e-05 2020-03-09T00:25:28.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.171 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.171 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866802+0000", 2020-03-09T00:25:28.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595 2020-03-09T00:25:28.171 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.171 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.171 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.172 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(osd.10.23:4 3.3s0 3:c607ec57:::smithi10428818-30:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e48)", 2020-03-09T00:25:28.172 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:28.172 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 30.361457288, 2020-03-09T00:25:28.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999, 2020-03-09T00:25:28.172 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:28.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8078999999999999e-05 2020-03-09T00:25:28.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789167+0000", 2020-03-09T00:25:28.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1499999999999998e-06 2020-03-09T00:25:28.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789172+0000", 2020-03-09T00:25:28.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6209999999999993e-06 2020-03-09T00:25:28.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789181+0000", 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4939999999999997e-06 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789185+0000", 2020-03-09T00:25:28.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5793e-05 2020-03-09T00:25:28.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789231+0000", 2020-03-09T00:25:28.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7570000000000002e-06 2020-03-09T00:25:28.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789235+0000", 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.99423119699999996 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783466+0000", 2020-03-09T00:25:28.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000227851 2020-03-09T00:25:28.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:28.181 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783694+0000", 2020-03-09T00:25:28.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056076839999999999 2020-03-09T00:25:28.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.181 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789302+0000", 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.182 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 53/52 ECSubWriteReply(tid=1, last_complete=34'2, committed=1, applied=1))", 2020-03-09T00:25:28.183 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:28.183 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.357288153999999, 2020-03-09T00:25:28.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215, 2020-03-09T00:25:28.183 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.183 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:28.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9154999999999999e-05 2020-03-09T00:25:28.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793357+0000", 2020-03-09T00:25:28.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4100000000000008e-06 2020-03-09T00:25:28.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793366+0000", 2020-03-09T00:25:28.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3783e-05 2020-03-09T00:25:28.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793380+0000", 2020-03-09T00:25:28.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6469999999999998e-06 2020-03-09T00:25:28.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793387+0000", 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012002081 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805389+0000", 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0139000000000001e-05 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805409+0000", 2020-03-09T00:25:28.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215 2020-03-09T00:25:28.189 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.189 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.189 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 53/52 ECSubWrite(tid=47, reqid=client.4751.0:355, at_version=53'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.29940341, 2020-03-09T00:25:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252, 2020-03-09T00:25:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.190 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.191 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:28.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00081411500000000002 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851997+0000", 2020-03-09T00:25:28.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0162999999999999e-05 2020-03-09T00:25:28.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852027+0000", 2020-03-09T00:25:28.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7360999999999999e-05 2020-03-09T00:25:28.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852045+0000", 2020-03-09T00:25:28.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2069000000000001e-05 2020-03-09T00:25:28.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852057+0000", 2020-03-09T00:25:28.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000131192 2020-03-09T00:25:28.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852188+0000", 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.517e-05 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852203+0000", 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099256910000000004 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862129+0000", 2020-03-09T00:25:28.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6758999999999997e-05 2020-03-09T00:25:28.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862176+0000", 2020-03-09T00:25:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252 2020-03-09T00:25:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.200 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=159, reqid=osd.12.0:159, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:28.200 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:28.200 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.330272053000002, 2020-03-09T00:25:28.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001, 2020-03-09T00:25:28.200 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.200 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.200 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:28.202 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1613999999999997e-05 2020-03-09T00:25:28.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.202 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.202 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820376+0000", 2020-03-09T00:25:28.202 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6792e-05 2020-03-09T00:25:28.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820403+0000", 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4728e-05 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820417+0000", 2020-03-09T00:25:28.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7910000000000005e-06 2020-03-09T00:25:28.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820425+0000", 2020-03-09T00:25:28.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010933589999999999 2020-03-09T00:25:28.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821519+0000", 2020-03-09T00:25:28.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3885e-05 2020-03-09T00:25:28.207 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.207 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.207 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.207 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821533+0000", 2020-03-09T00:25:28.208 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011394939999999999 2020-03-09T00:25:28.208 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.209 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.209 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.209 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832927+0000", 2020-03-09T00:25:28.209 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6371e-05 2020-03-09T00:25:28.209 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.209 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.209 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832974+0000", 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.210 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=5, reqid=client.4751.0:485, at_version=59'26, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:28.211 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:28.211 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.329630117000001, 2020-03-09T00:25:28.211 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083, 2020-03-09T00:25:28.211 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.211 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.211 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.211 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:28.212 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4487000000000002e-05 2020-03-09T00:25:28.213 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.213 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.213 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821011+0000", 2020-03-09T00:25:28.213 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9616e-05 2020-03-09T00:25:28.213 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821030+0000", 2020-03-09T00:25:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4766000000000001e-05 2020-03-09T00:25:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821045+0000", 2020-03-09T00:25:28.215 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.789e-06 2020-03-09T00:25:28.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.215 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.215 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.215 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821051+0000", 2020-03-09T00:25:28.215 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010220629999999999 2020-03-09T00:25:28.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.216 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.216 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.216 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822073+0000", 2020-03-09T00:25:28.216 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1511999999999999e-05 2020-03-09T00:25:28.216 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.216 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.217 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.217 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822085+0000", 2020-03-09T00:25:28.217 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010894232 2020-03-09T00:25:28.217 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.217 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.217 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.217 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832979+0000", 2020-03-09T00:25:28.218 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:28.218 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.218 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.218 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.218 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833022+0000", 2020-03-09T00:25:28.218 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083 2020-03-09T00:25:28.218 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.219 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.219 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.219 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.219 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.219 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=6, reqid=client.4751.0:486, at_version=59'27, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:28.219 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:28.220 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.329485235, 2020-03-09T00:25:28.220 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978, 2020-03-09T00:25:28.220 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.220 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.220 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.220 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.220 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.221 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:28.221 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.221 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.221 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.221 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.221 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:28.221 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4271000000000003e-05 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821155+0000", 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8627e-05 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.222 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.223 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821174+0000", 2020-03-09T00:25:28.223 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4825e-05 2020-03-09T00:25:28.223 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.223 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.223 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.223 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821189+0000", 2020-03-09T00:25:28.223 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4070000000000001e-06 2020-03-09T00:25:28.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.224 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.224 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821194+0000", 2020-03-09T00:25:28.224 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039590839999999999 2020-03-09T00:25:28.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825153+0000", 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3074999999999999e-05 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825166+0000", 2020-03-09T00:25:28.225 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078593580000000003 2020-03-09T00:25:28.226 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.226 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.226 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.226 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833026+0000", 2020-03-09T00:25:28.226 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2330999999999999e-05 2020-03-09T00:25:28.226 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.226 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.227 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.227 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833058+0000", 2020-03-09T00:25:28.227 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978 2020-03-09T00:25:28.227 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.227 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.227 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.227 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=59'43, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.329280213000001, 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525, 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.228 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.229 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.229 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.229 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:28.229 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.229 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.229 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.229 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.230 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:28.230 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6616999999999997e-05 2020-03-09T00:25:28.230 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.230 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.230 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.230 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821363+0000", 2020-03-09T00:25:28.230 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1061e-05 2020-03-09T00:25:28.231 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.231 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.231 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.231 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821384+0000", 2020-03-09T00:25:28.231 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5315000000000002e-05 2020-03-09T00:25:28.231 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.231 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821399+0000", 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0929999999999999e-06 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821405+0000", 2020-03-09T00:25:28.232 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003024559 2020-03-09T00:25:28.233 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.233 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.233 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824430+0000", 2020-03-09T00:25:28.233 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2744e-05 2020-03-09T00:25:28.233 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824443+0000", 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010025054 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834468+0000", 2020-03-09T00:25:28.234 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0082e-05 2020-03-09T00:25:28.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.235 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.235 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.235 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834508+0000", 2020-03-09T00:25:28.235 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525 2020-03-09T00:25:28.235 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.235 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=7, reqid=client.4751.0:487, at_version=59'28, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.310937622000001, 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916, 2020-03-09T00:25:28.236 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.238 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.238 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:28.238 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7596999999999998e-05 2020-03-09T00:25:28.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.238 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.239 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839706+0000", 2020-03-09T00:25:28.239 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5558000000000002e-05 2020-03-09T00:25:28.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.239 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.239 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839732+0000", 2020-03-09T00:25:28.240 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0251999999999999e-05 2020-03-09T00:25:28.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.240 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.240 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839752+0000", 2020-03-09T00:25:28.240 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0780000000000002e-06 2020-03-09T00:25:28.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.241 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.241 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839761+0000", 2020-03-09T00:25:28.241 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010019800000000001 2020-03-09T00:25:28.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.242 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.242 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.242 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839861+0000", 2020-03-09T00:25:28.242 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9032e-05 2020-03-09T00:25:28.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.242 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839881+0000", 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014066339000000001 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853947+0000", 2020-03-09T00:25:28.243 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000210862 2020-03-09T00:25:28.244 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.244 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.244 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.244 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854158+0000", 2020-03-09T00:25:28.244 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916 2020-03-09T00:25:28.244 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.244 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=164, reqid=client.4751.0:490, at_version=59'45, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.310719191, 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001, 2020-03-09T00:25:28.245 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.246 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.246 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.246 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.246 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.246 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:28.246 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.246 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.247 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.247 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.247 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:28.247 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000195997 2020-03-09T00:25:28.247 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.247 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.247 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.248 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840063+0000", 2020-03-09T00:25:28.248 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7005e-05 2020-03-09T00:25:28.248 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.248 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.248 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.248 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840090+0000", 2020-03-09T00:25:28.248 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6943999999999999e-05 2020-03-09T00:25:28.249 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.249 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.249 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.249 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840107+0000", 2020-03-09T00:25:28.249 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7489999999999998e-06 2020-03-09T00:25:28.249 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.249 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.250 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.250 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840115+0000", 2020-03-09T00:25:28.250 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019725999999999999 2020-03-09T00:25:28.250 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.250 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.250 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.251 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840312+0000", 2020-03-09T00:25:28.251 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4409e-05 2020-03-09T00:25:28.251 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.251 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.251 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.251 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840327+0000", 2020-03-09T00:25:28.251 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018701885000000001 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859028+0000", 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7057e-05 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.252 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.253 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859076+0000", 2020-03-09T00:25:28.253 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001 2020-03-09T00:25:28.253 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.253 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.253 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.253 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.253 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.307210411, 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002, 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.254 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.255 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.255 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:28.255 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.255 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.255 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.255 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.255 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012568360000000001 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844633+0000", 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0247999999999999e-05 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.256 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.257 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.257 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844663+0000", 2020-03-09T00:25:28.257 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6184e-05 2020-03-09T00:25:28.257 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.257 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.257 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.257 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844679+0000", 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8560000000000001e-06 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844686+0000", 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8074e-05 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.258 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.259 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.259 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844744+0000", 2020-03-09T00:25:28.259 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9119000000000001e-05 2020-03-09T00:25:28.259 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.259 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.259 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.259 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844763+0000", 2020-03-09T00:25:28.260 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014486196999999999 2020-03-09T00:25:28.260 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.260 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.260 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.260 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859250+0000", 2020-03-09T00:25:28.261 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115911 2020-03-09T00:25:28.261 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.261 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.261 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.262 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859365+0000", 2020-03-09T00:25:28.262 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002 2020-03-09T00:25:28.262 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.262 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.262 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.262 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.263 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.263 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=90, reqid=client.4751.0:499, at_version=59'56, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:28.263 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:28.263 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.301504639000001, 2020-03-09T00:25:28.263 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999, 2020-03-09T00:25:28.263 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.263 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.264 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.265 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:28.265 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050415699999999998 2020-03-09T00:25:28.265 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.265 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.265 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.265 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849586+0000", 2020-03-09T00:25:28.265 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9476e-05 2020-03-09T00:25:28.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.266 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.266 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.266 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849615+0000", 2020-03-09T00:25:28.266 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8382000000000001e-05 2020-03-09T00:25:28.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.267 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.267 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849634+0000", 2020-03-09T00:25:28.267 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0439999999999996e-06 2020-03-09T00:25:28.267 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.268 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.268 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849642+0000", 2020-03-09T00:25:28.268 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018423140000000001 2020-03-09T00:25:28.268 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.269 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.269 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.269 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851484+0000", 2020-03-09T00:25:28.269 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3304e-05 2020-03-09T00:25:28.269 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.269 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.270 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.270 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851497+0000", 2020-03-09T00:25:28.270 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097988120000000005 2020-03-09T00:25:28.270 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.270 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.270 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.270 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861296+0000", 2020-03-09T00:25:28.271 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1790999999999999e-05 2020-03-09T00:25:28.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.271 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.271 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.271 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861348+0000", 2020-03-09T00:25:28.271 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999 2020-03-09T00:25:28.271 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:605 3.3s0 3:c2119ac7:::smithi10428818-4:head [read 0~573440,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801341+0000", 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.3492457679999994, 2020-03-09T00:25:28.272 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098273839999999998, 2020-03-09T00:25:28.273 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.273 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:28.273 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:28.273 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:28.273 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.273 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 605 2020-03-09T00:25:28.273 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.274 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.274 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.274 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.274 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801341+0000", 2020-03-09T00:25:28.274 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.274 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.274 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801341+0000", 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5411999999999998e-05 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801386+0000", 2020-03-09T00:25:28.275 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5180000000000001e-06 2020-03-09T00:25:28.276 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.276 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.276 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.276 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801393+0000", 2020-03-09T00:25:28.276 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3062e-05 2020-03-09T00:25:28.276 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.277 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.278 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.278 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801406+0000", 2020-03-09T00:25:28.278 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1629999999999999e-06 2020-03-09T00:25:28.278 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.279 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.279 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.279 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801413+0000", 2020-03-09T00:25:28.279 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3575e-05 2020-03-09T00:25:28.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.280 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.280 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.280 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801476+0000", 2020-03-09T00:25:28.280 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000138405 2020-03-09T00:25:28.281 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.281 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.281 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:28.281 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801615+0000", 2020-03-09T00:25:28.281 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0095532489999999998 2020-03-09T00:25:28.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.282 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.282 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.282 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.811168+0000", 2020-03-09T00:25:28.282 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098273839999999998 2020-03-09T00:25:28.282 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.283 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.283 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.283 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.283 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:606 3.3s0 3:c2119ac7:::smithi10428818-4:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:28.284 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801456+0000", 2020-03-09T00:25:28.284 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.3491303469999991, 2020-03-09T00:25:28.284 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010436978, 2020-03-09T00:25:28.284 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.284 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:28.284 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:28.285 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:28.285 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.285 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 606 2020-03-09T00:25:28.285 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.285 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.286 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.286 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.286 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801456+0000", 2020-03-09T00:25:28.286 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.286 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.286 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.286 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.287 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801456+0000", 2020-03-09T00:25:28.287 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9487000000000003e-05 2020-03-09T00:25:28.287 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.287 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.287 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.288 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801496+0000", 2020-03-09T00:25:28.288 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7809999999999999e-06 2020-03-09T00:25:28.288 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.288 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.289 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.289 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801499+0000", 2020-03-09T00:25:28.289 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1799999999999996e-06 2020-03-09T00:25:28.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.289 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.290 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.290 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801508+0000", 2020-03-09T00:25:28.290 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9500000000000001e-06 2020-03-09T00:25:28.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.290 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.290 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.291 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801510+0000", 2020-03-09T00:25:28.291 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076127400000000004 2020-03-09T00:25:28.291 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.291 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.291 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.291 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.802272+0000", 2020-03-09T00:25:28.291 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000122903 2020-03-09T00:25:28.292 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.292 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.292 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:28.292 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.802395+0000", 2020-03-09T00:25:28.292 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094984030000000007 2020-03-09T00:25:28.292 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.292 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.811893+0000", 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010436978 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.293 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.294 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:607 3.3s0 3:c2119ac7:::smithi10428818-4:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:28.294 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:28.294 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.3490364489999997, 2020-03-09T00:25:28.295 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011089972999999999, 2020-03-09T00:25:28.295 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.295 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:28.295 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:28.296 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:28.296 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.296 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 607 2020-03-09T00:25:28.296 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.296 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.297 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.297 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.297 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:28.297 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.297 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.297 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.297 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.298 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:28.298 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7959e-05 2020-03-09T00:25:28.298 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.298 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.298 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.298 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801588+0000", 2020-03-09T00:25:28.298 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.676e-06 2020-03-09T00:25:28.299 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.299 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.299 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801592+0000", 2020-03-09T00:25:28.299 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3799999999999999e-06 2020-03-09T00:25:28.299 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801598+0000", 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5610000000000004e-06 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801605+0000", 2020-03-09T00:25:28.300 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015221259999999999 2020-03-09T00:25:28.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.301 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.301 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.803127+0000", 2020-03-09T00:25:28.301 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012211399999999999 2020-03-09T00:25:28.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.803249+0000", 2020-03-09T00:25:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0093911570000000007 2020-03-09T00:25:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.812640+0000", 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011089972999999999 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:608 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:28.303 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:28.304 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.3489512129999994, 2020-03-09T00:25:28.304 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013652041, 2020-03-09T00:25:28.304 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.304 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:28.304 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:28.304 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:28.304 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.305 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 608 2020-03-09T00:25:28.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.305 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.305 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.305 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.305 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:28.305 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.306 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.306 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.306 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.306 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:28.306 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5256999999999997e-05 2020-03-09T00:25:28.306 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.306 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.307 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.307 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801680+0000", 2020-03-09T00:25:28.307 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3590000000000002e-06 2020-03-09T00:25:28.307 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.307 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.307 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.307 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801688+0000", 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1552999999999999e-05 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801699+0000", 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3519999999999997e-06 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.308 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801705+0000", 2020-03-09T00:25:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003598459 2020-03-09T00:25:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.805303+0000", 2020-03-09T00:25:28.310 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000173425 2020-03-09T00:25:28.310 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.310 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.310 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:28.310 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.805477+0000", 2020-03-09T00:25:28.310 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098106359999999993 2020-03-09T00:25:28.310 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.815287+0000", 2020-03-09T00:25:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013652041 2020-03-09T00:25:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:609 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.3488316250000008, 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013941073999999999, 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.312 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:28.313 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:28.313 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:28.313 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.313 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 609 2020-03-09T00:25:28.313 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.313 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.313 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.314 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.314 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:28.314 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.314 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.314 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.314 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.314 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0500000000000006e-05 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801825+0000", 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.614e-06 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.315 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.316 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.316 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801831+0000", 2020-03-09T00:25:28.316 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5819999999999998e-06 2020-03-09T00:25:28.316 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.316 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.316 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.316 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801837+0000", 2020-03-09T00:25:28.317 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.619e-06 2020-03-09T00:25:28.317 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.317 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.317 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.317 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801841+0000", 2020-03-09T00:25:28.317 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042523969999999998 2020-03-09T00:25:28.317 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.318 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.318 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.318 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806094+0000", 2020-03-09T00:25:28.318 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011631799999999999 2020-03-09T00:25:28.318 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.318 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.318 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:28.319 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806210+0000", 2020-03-09T00:25:28.319 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094860440000000008 2020-03-09T00:25:28.319 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.319 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.319 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.319 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.815696+0000", 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013941073999999999 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:610 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:28.320 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:28.321 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 9.3486942069999994, 2020-03-09T00:25:28.321 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014287645, 2020-03-09T00:25:28.321 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.321 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:28.321 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:28.321 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:28.321 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 610 2020-03-09T00:25:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:28.322 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.323 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.323 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.323 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.323 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:28.323 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5413000000000005e-05 2020-03-09T00:25:28.323 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.323 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801968+0000", 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4571e-05 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801982+0000", 2020-03-09T00:25:28.324 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3568e-05 2020-03-09T00:25:28.325 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.325 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.325 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.325 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801996+0000", 2020-03-09T00:25:28.325 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4940000000000003e-06 2020-03-09T00:25:28.325 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.325 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.802001+0000", 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046307759999999996 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806632+0000", 2020-03-09T00:25:28.326 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011677599999999999 2020-03-09T00:25:28.327 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.327 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.327 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:28.327 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806749+0000", 2020-03-09T00:25:28.327 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094310469999999997 2020-03-09T00:25:28.327 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.327 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.328 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.328 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.816180+0000", 2020-03-09T00:25:28.328 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014287645 2020-03-09T00:25:28.328 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.328 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.328 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.328 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.4s6 75/67 ECSubWrite(tid=93, reqid=client.4751.0:637, at_version=75'8, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.3408190470000001, 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021185215, 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.329 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:28.330 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.330 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:28.330 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:28.330 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:28.330 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.330 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.331 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:28.331 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:28.331 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021834200000000002 2020-03-09T00:25:28.331 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.331 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.331 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:28.332 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811951+0000", 2020-03-09T00:25:28.332 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8116999999999999e-05 2020-03-09T00:25:28.332 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.332 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.332 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:28.332 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811979+0000", 2020-03-09T00:25:28.332 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3502e-05 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811992+0000", 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9779999999999999e-06 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.333 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.334 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811999+0000", 2020-03-09T00:25:28.334 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6839999999999998e-05 2020-03-09T00:25:28.334 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.334 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.334 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:28.334 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.812056+0000", 2020-03-09T00:25:28.334 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5362e-05 2020-03-09T00:25:28.335 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.335 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.335 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.335 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.812072+0000", 2020-03-09T00:25:28.335 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01883961 2020-03-09T00:25:28.335 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.335 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.830911+0000", 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1386e-05 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.830953+0000", 2020-03-09T00:25:28.336 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021185215 2020-03-09T00:25:28.337 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.337 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.337 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.337 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:28.337 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:28.337 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:28.339 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:28.340 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:28.340 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:28.340 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:28.340 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:28.340 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:28.340 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:28.340 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:28.341 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:25:28.341 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:28.301+0000 7f9045d9b700 0 -- 172.21.15.148:0/519983069 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7f9048188bb0 msgr2=0x7f904818afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-03-09T00:25:28.342 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:28.337+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:28.440 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:28.438+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:28.457 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:28.458 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:28.458 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:28.458 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:28.466 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:25:28.523 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:28.524 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.524 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.524 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.524 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.524 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.525 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.526 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.526 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.526 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.526 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:28.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:28.574 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:28.573+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:28.582 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:28.582 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:28.582 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:28.582 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:28.583 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:28.591 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:25:28.641 INFO:tasks.daemonwatchdog.daemon_watchdog:OSDThrasher failed 2020-03-09T00:25:28.641 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2020-03-09T00:25:28.642 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/daemonwatchdog.py", line 37, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/daemonwatchdog.py", line 113, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks/daemonwatchdog.py", line 53, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 253, in __getattr__ raise AttributeError(name) AttributeError: mounts 2020-03-09T00:25:28.646 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:28.641+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:28.708 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:28.709 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:28.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:28.709 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:28.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.709 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:207 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 5] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:28.709 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 40.849267945999998, 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001, 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.710 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207 2020-03-09T00:25:28.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.711 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:28.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7176999999999996e-05 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856995+0000", 2020-03-09T00:25:28.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7161999999999999e-05 2020-03-09T00:25:28.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857013+0000", 2020-03-09T00:25:28.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2879e-05 2020-03-09T00:25:28.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857026+0000", 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6669999999999996e-06 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857033+0000", 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6767999999999997e-05 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857080+0000", 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4165999999999994e-05 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857154+0000", 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068311889999999997 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.863985+0000", 2020-03-09T00:25:28.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041278249999999999 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868113+0000", 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001971132 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870084+0000", 2020-03-09T00:25:28.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113266 2020-03-09T00:25:28.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870198+0000", 2020-03-09T00:25:28.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:208 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:28.719 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 40.8490982, 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001, 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 208 2020-03-09T00:25:28.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.721 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:28.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:28.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5717000000000003e-05 2020-03-09T00:25:28.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857134+0000", 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3659999999999997e-06 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857140+0000", 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2546e-05 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857153+0000", 2020-03-09T00:25:28.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7190000000000001e-06 2020-03-09T00:25:28.724 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.724 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.724 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.724 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857157+0000", 2020-03-09T00:25:28.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134509 2020-03-09T00:25:28.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.725 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.725 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.725 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857292+0000", 2020-03-09T00:25:28.725 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5519999999999999e-05 2020-03-09T00:25:28.725 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857327+0000", 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010666132 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.726 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867994+0000", 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110682 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868104+0000", 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00321515 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.727 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.728 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.728 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871319+0000", 2020-03-09T00:25:28.728 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011457 2020-03-09T00:25:28.728 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.728 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871434+0000", 2020-03-09T00:25:28.729 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013121699999999999 2020-03-09T00:25:28.729 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.729 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.729 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.729 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871565+0000", 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 44/32 ECSubWrite(tid=44, reqid=osd.13.0:44, at_version=44'11, trim_to=0'0, roll_forward_to=42'9))", 2020-03-09T00:25:28.730 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:28.731 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 34.857044969, 2020-03-09T00:25:28.731 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002, 2020-03-09T00:25:28.731 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.731 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.731 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.731 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:28.732 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.732 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.732 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.732 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.732 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:28.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8740999999999999e-05 2020-03-09T00:25:28.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.733 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.733 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.733 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849200+0000", 2020-03-09T00:25:28.733 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2819e-05 2020-03-09T00:25:28.733 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.734 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.734 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849223+0000", 2020-03-09T00:25:28.734 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5366000000000002e-05 2020-03-09T00:25:28.734 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.734 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849238+0000", 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7410000000000003e-06 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849245+0000", 2020-03-09T00:25:28.735 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5445999999999996e-05 2020-03-09T00:25:28.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.736 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.736 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849310+0000", 2020-03-09T00:25:28.736 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3651e-05 2020-03-09T00:25:28.736 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.736 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849324+0000", 2020-03-09T00:25:28.737 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057096769999999998 2020-03-09T00:25:28.737 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.737 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.737 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.737 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855034+0000", 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001136 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855147+0000", 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.738 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 34.855903384000001, 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998, 2020-03-09T00:25:28.739 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.740 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.740 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.740 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.740 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.740 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:28.740 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.740 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.741 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:28.741 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018406799999999999 2020-03-09T00:25:28.741 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.741 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.741 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850467+0000", 2020-03-09T00:25:28.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2490999999999999e-05 2020-03-09T00:25:28.742 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.742 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.742 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.742 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850489+0000", 2020-03-09T00:25:28.742 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6807999999999999e-05 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9419999999999995e-06 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.743 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850514+0000", 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088484100000000003 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851399+0000", 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2279999999999999e-05 2020-03-09T00:25:28.744 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851411+0000", 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083268249999999995 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.745 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859738+0000", 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2222999999999998e-05 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859790+0000", 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.746 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.888788695999999, 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001, 2020-03-09T00:25:28.747 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.748 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069548899999999998 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818093+0000", 2020-03-09T00:25:28.749 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6591999999999999e-05 2020-03-09T00:25:28.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.750 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.750 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.750 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818120+0000", 2020-03-09T00:25:28.750 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5192e-05 2020-03-09T00:25:28.750 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818135+0000", 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4989999999999997e-06 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.751 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818142+0000", 2020-03-09T00:25:28.752 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4997000000000002e-05 2020-03-09T00:25:28.752 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.752 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818197+0000", 2020-03-09T00:25:28.753 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3164e-05 2020-03-09T00:25:28.753 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.753 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.753 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.753 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818210+0000", 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092824440000000008 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827493+0000", 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2645999999999997e-05 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.754 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.755 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.755 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827546+0000", 2020-03-09T00:25:28.755 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001 2020-03-09T00:25:28.755 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.755 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.755 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.755 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s6 53/52 ECSubWrite(tid=3, reqid=client.4751.0:328, at_version=0'0, trim_to=0'0, roll_forward_to=53'17))", 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.809490+0000", 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.896696681000002, 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054575800000000001, 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.756 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809490+0000", 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.757 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809490+0000", 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7173000000000002e-05 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809557+0000", 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1239000000000001e-05 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.758 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.759 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.759 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809578+0000", 2020-03-09T00:25:28.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5971e-05 2020-03-09T00:25:28.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.760 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.760 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.760 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809594+0000", 2020-03-09T00:25:28.760 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6680000000000004e-06 2020-03-09T00:25:28.760 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.809601+0000", 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002699417 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.761 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812300+0000", 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6064000000000001e-05 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812316+0000", 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025918120000000002 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.762 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.814908+0000", 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9236000000000002e-05 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.814947+0000", 2020-03-09T00:25:28.763 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054575800000000001 2020-03-09T00:25:28.764 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.764 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.764 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.764 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.764 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.764 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s3 53/48 ECSubWrite(tid=30, reqid=client.4751.0:346, at_version=53'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:28.764 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.879290360999999, 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059893999999999998, 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.765 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000139486 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.766 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827035+0000", 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5199999999999999e-05 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827061+0000", 2020-03-09T00:25:28.767 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376999999999999e-05 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827076+0000", 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8279999999999997e-06 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.768 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827083+0000", 2020-03-09T00:25:28.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050597700000000001 2020-03-09T00:25:28.769 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.769 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.769 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.769 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827589+0000", 2020-03-09T00:25:28.769 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2581e-05 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827601+0000", 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052374839999999997 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.770 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832839+0000", 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6467000000000001e-05 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832885+0000", 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059893999999999998 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.771 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.772 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.772 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.772 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.774 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s3 53/48 ECSubWrite(tid=31, reqid=client.4751.0:347, at_version=53'19, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:28.774 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.834371+0000", 2020-03-09T00:25:28.774 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.871815178999999, 2020-03-09T00:25:28.774 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054921900000000001, 2020-03-09T00:25:28.774 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834371+0000", 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.775 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834371+0000", 2020-03-09T00:25:28.776 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014995699999999999 2020-03-09T00:25:28.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.776 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.776 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.776 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834521+0000", 2020-03-09T00:25:28.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0777000000000001e-05 2020-03-09T00:25:28.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.777 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.777 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.777 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834542+0000", 2020-03-09T00:25:28.777 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9723000000000001e-05 2020-03-09T00:25:28.777 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834562+0000", 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.359e-05 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.778 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834575+0000", 2020-03-09T00:25:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8232000000000007e-05 2020-03-09T00:25:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834643+0000", 2020-03-09T00:25:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3478999999999999e-05 2020-03-09T00:25:28.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834657+0000", 2020-03-09T00:25:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005132803 2020-03-09T00:25:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.780 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.839790+0000", 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3628999999999996e-05 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.839863+0000", 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054921900000000001 2020-03-09T00:25:28.781 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.782 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.782 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.782 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.782 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.782 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 54/48 ECSubWrite(tid=8, reqid=client.4751.0:376, at_version=54'17, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:28.782 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:28.782 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.896775546000001, 2020-03-09T00:25:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002, 2020-03-09T00:25:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.783 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000509736 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.784 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809920+0000", 2020-03-09T00:25:28.785 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8026999999999999e-05 2020-03-09T00:25:28.785 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.785 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.785 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.785 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809938+0000", 2020-03-09T00:25:28.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829e-05 2020-03-09T00:25:28.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.786 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.786 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.786 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809954+0000", 2020-03-09T00:25:28.786 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8799999999999997e-06 2020-03-09T00:25:28.786 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.787 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809963+0000", 2020-03-09T00:25:28.787 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3013000000000003e-05 2020-03-09T00:25:28.787 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.787 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.787 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810006+0000", 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2413e-05 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810019+0000", 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067421620000000003 2020-03-09T00:25:28.788 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.789 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816761+0000", 2020-03-09T00:25:28.789 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3118999999999997e-05 2020-03-09T00:25:28.789 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.789 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.789 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.790 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816814+0000", 2020-03-09T00:25:28.790 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002 2020-03-09T00:25:28.790 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.790 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.790 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.790 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.790 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.791 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 59/48 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=59'30, trim_to=0'0, roll_forward_to=58'28))", 2020-03-09T00:25:28.791 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:28.791 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.893581874999999, 2020-03-09T00:25:28.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067353739999999997, 2020-03-09T00:25:28.791 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.791 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.791 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.792 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:28.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4576000000000001e-05 2020-03-09T00:25:28.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812659+0000", 2020-03-09T00:25:28.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0911999999999998e-05 2020-03-09T00:25:28.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812690+0000", 2020-03-09T00:25:28.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4375999999999999e-05 2020-03-09T00:25:28.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812704+0000", 2020-03-09T00:25:28.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2119999999999997e-06 2020-03-09T00:25:28.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812711+0000", 2020-03-09T00:25:28.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016912599999999999 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812881+0000", 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.377e-05 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812894+0000", 2020-03-09T00:25:28.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064052889999999998 2020-03-09T00:25:28.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819300+0000", 2020-03-09T00:25:28.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0113e-05 2020-03-09T00:25:28.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819340+0000", 2020-03-09T00:25:28.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067353739999999997 2020-03-09T00:25:28.799 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.799 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.799 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.799 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:507 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 54] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:28.799 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.801496797999999, 2020-03-09T00:25:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001, 2020-03-09T00:25:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.801 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 507 2020-03-09T00:25:28.801 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.801 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:28.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.802 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.802 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:28.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4511999999999997e-05 2020-03-09T00:25:28.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.803 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.803 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.803 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904734+0000", 2020-03-09T00:25:28.803 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3980000000000004e-06 2020-03-09T00:25:28.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904740+0000", 2020-03-09T00:25:28.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4293e-05 2020-03-09T00:25:28.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904755+0000", 2020-03-09T00:25:28.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4449999999999996e-06 2020-03-09T00:25:28.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904763+0000", 2020-03-09T00:25:28.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0075999999999999e-05 2020-03-09T00:25:28.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904823+0000", 2020-03-09T00:25:28.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8325999999999994e-05 2020-03-09T00:25:28.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904911+0000", 2020-03-09T00:25:28.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091150910000000005 2020-03-09T00:25:28.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914027+0000", 2020-03-09T00:25:28.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097858200000000006 2020-03-09T00:25:28.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923812+0000", 2020-03-09T00:25:28.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072064440000000002 2020-03-09T00:25:28.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931019+0000", 2020-03-09T00:25:28.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6091000000000001e-05 2020-03-09T00:25:28.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.812 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.812 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931085+0000", 2020-03-09T00:25:28.812 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001 2020-03-09T00:25:28.813 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.813 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.813 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.813 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.814 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:508 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:28.814 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:28.814 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.801384729999999, 2020-03-09T00:25:28.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261, 2020-03-09T00:25:28.815 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.815 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:28.815 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.816 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.816 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.816 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 508 2020-03-09T00:25:28.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.816 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:28.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:28.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4413000000000002e-05 2020-03-09T00:25:28.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904826+0000", 2020-03-09T00:25:28.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1389999999999997e-06 2020-03-09T00:25:28.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904830+0000", 2020-03-09T00:25:28.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3339999999999996e-06 2020-03-09T00:25:28.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.822 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904837+0000", 2020-03-09T00:25:28.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8739999999999999e-06 2020-03-09T00:25:28.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.823 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.823 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904840+0000", 2020-03-09T00:25:28.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000222303 2020-03-09T00:25:28.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905063+0000", 2020-03-09T00:25:28.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5607999999999999e-05 2020-03-09T00:25:28.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:28.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905128+0000", 2020-03-09T00:25:28.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018331841000000001 2020-03-09T00:25:28.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923460+0000", 2020-03-09T00:25:28.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012596800000000001 2020-03-09T00:25:28.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:28.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923586+0000", 2020-03-09T00:25:28.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0084087399999999996 2020-03-09T00:25:28.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931995+0000", 2020-03-09T00:25:28.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010665 2020-03-09T00:25:28.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932101+0000", 2020-03-09T00:25:28.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012239099999999999 2020-03-09T00:25:28.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932224+0000", 2020-03-09T00:25:28.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261 2020-03-09T00:25:28.832 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.833 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.833 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.834 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 64/32 ECSubWrite(tid=119, reqid=osd.8.0:118, at_version=0'0, trim_to=0'0, roll_forward_to=64'83))", 2020-03-09T00:25:28.834 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:13.783969+0000", 2020-03-09T00:25:28.834 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 14.922216999, 2020-03-09T00:25:28.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054809639999999996, 2020-03-09T00:25:28.835 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.835 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.835 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.783969+0000", 2020-03-09T00:25:28.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.783969+0000", 2020-03-09T00:25:28.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0179000000000001e-05 2020-03-09T00:25:28.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784029+0000", 2020-03-09T00:25:28.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.307e-05 2020-03-09T00:25:28.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784042+0000", 2020-03-09T00:25:28.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5985e-05 2020-03-09T00:25:28.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784058+0000", 2020-03-09T00:25:28.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9649999999999997e-06 2020-03-09T00:25:28.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.784067+0000", 2020-03-09T00:25:28.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029749799999999999 2020-03-09T00:25:28.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787042+0000", 2020-03-09T00:25:28.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5451000000000002e-05 2020-03-09T00:25:28.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.844 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.845 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787058+0000", 2020-03-09T00:25:28.845 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023299979999999998 2020-03-09T00:25:28.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.845 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.789388+0000", 2020-03-09T00:25:28.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2335999999999994e-05 2020-03-09T00:25:28.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.789450+0000", 2020-03-09T00:25:28.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054809639999999996 2020-03-09T00:25:28.847 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.848 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.848 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.849 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 68/48 ECSubWrite(tid=38, reqid=client.4751.0:589, at_version=68'45, trim_to=0'0, roll_forward_to=67'41))", 2020-03-09T00:25:28.849 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.987799+0000", 2020-03-09T00:25:28.849 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.718386998, 2020-03-09T00:25:28.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066719780000000003, 2020-03-09T00:25:28.850 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.850 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:28.850 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.987799+0000", 2020-03-09T00:25:28.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.987799+0000", 2020-03-09T00:25:28.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00034108499999999998 2020-03-09T00:25:28.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.854 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988140+0000", 2020-03-09T00:25:28.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9864000000000002e-05 2020-03-09T00:25:28.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.854 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988160+0000", 2020-03-09T00:25:28.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5350999999999999e-05 2020-03-09T00:25:28.855 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.855 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.855 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.855 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988176+0000", 2020-03-09T00:25:28.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8819999999999998e-06 2020-03-09T00:25:28.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988181+0000", 2020-03-09T00:25:28.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5544999999999999e-05 2020-03-09T00:25:28.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988217+0000", 2020-03-09T00:25:28.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.294e-06 2020-03-09T00:25:28.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988224+0000", 2020-03-09T00:25:28.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061940880000000004 2020-03-09T00:25:28.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.994418+0000", 2020-03-09T00:25:28.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2868999999999998e-05 2020-03-09T00:25:28.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.994471+0000", 2020-03-09T00:25:28.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066719780000000003 2020-03-09T00:25:28.859 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.860 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.860 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.860 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:598 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:28.860 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:28.860 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.9176487899999994, 2020-03-09T00:25:28.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014064482999999999, 2020-03-09T00:25:28.861 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.861 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.861 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.861 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 598 2020-03-09T00:25:28.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.862 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:28.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.863 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:28.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4369999999999999e-05 2020-03-09T00:25:28.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788562+0000", 2020-03-09T00:25:28.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8599999999999998e-06 2020-03-09T00:25:28.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788568+0000", 2020-03-09T00:25:28.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5980000000000003e-06 2020-03-09T00:25:28.866 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788577+0000", 2020-03-09T00:25:28.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.955e-06 2020-03-09T00:25:28.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788582+0000", 2020-03-09T00:25:28.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5593000000000002e-05 2020-03-09T00:25:28.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788628+0000", 2020-03-09T00:25:28.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3660000000000003e-06 2020-03-09T00:25:28.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:28.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788635+0000", 2020-03-09T00:25:28.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019670759999999999 2020-03-09T00:25:28.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.790602+0000", 2020-03-09T00:25:28.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0575999999999995e-05 2020-03-09T00:25:28.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.790693+0000", 2020-03-09T00:25:28.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00085902600000000004 2020-03-09T00:25:28.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.791552+0000", 2020-03-09T00:25:28.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052594449999999997 2020-03-09T00:25:28.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796811+0000", 2020-03-09T00:25:28.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056778230000000002 2020-03-09T00:25:28.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802489+0000", 2020-03-09T00:25:28.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011279499999999999 2020-03-09T00:25:28.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802602+0000", 2020-03-09T00:25:28.874 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014064482999999999 2020-03-09T00:25:28.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.874 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.874 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:599 3.as0 3:51058371:::smithi10428818-28:head [rollback 1a] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:28.875 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:28.875 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.9175603690000003, 2020-03-09T00:25:28.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014184085000000001, 2020-03-09T00:25:28.875 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.875 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.876 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.876 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 599 2020-03-09T00:25:28.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.877 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:28.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:28.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8149999999999998e-05 2020-03-09T00:25:28.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788674+0000", 2020-03-09T00:25:28.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7279999999999998e-06 2020-03-09T00:25:28.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788681+0000", 2020-03-09T00:25:28.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.347e-05 2020-03-09T00:25:28.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788694+0000", 2020-03-09T00:25:28.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8910000000000003e-06 2020-03-09T00:25:28.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.883 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.884 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.884 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788699+0000", 2020-03-09T00:25:28.884 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2481999999999998e-05 2020-03-09T00:25:28.885 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.885 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.885 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.885 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788732+0000", 2020-03-09T00:25:28.886 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4740000000000002e-06 2020-03-09T00:25:28.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:28.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788738+0000", 2020-03-09T00:25:28.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044600949999999999 2020-03-09T00:25:28.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.793198+0000", 2020-03-09T00:25:28.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000122469 2020-03-09T00:25:28.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.793321+0000", 2020-03-09T00:25:28.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081692400000000001 2020-03-09T00:25:28.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794138+0000", 2020-03-09T00:25:28.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026755360000000001 2020-03-09T00:25:28.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796813+0000", 2020-03-09T00:25:28.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058884480000000001 2020-03-09T00:25:28.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802702+0000", 2020-03-09T00:25:28.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108418 2020-03-09T00:25:28.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.894 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802810+0000", 2020-03-09T00:25:28.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014184085000000001 2020-03-09T00:25:28.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.894 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.895 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.895 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:600 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:28.895 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:28.896 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.9174368709999996, 2020-03-09T00:25:28.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016073481000000001, 2020-03-09T00:25:28.896 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.896 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.897 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.897 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.897 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.897 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 600 2020-03-09T00:25:28.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.898 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:28.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:28.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3720999999999998e-05 2020-03-09T00:25:28.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788783+0000", 2020-03-09T00:25:28.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5499999999999999e-06 2020-03-09T00:25:28.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788787+0000", 2020-03-09T00:25:28.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0769e-05 2020-03-09T00:25:28.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.903 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788797+0000", 2020-03-09T00:25:28.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.157e-06 2020-03-09T00:25:28.903 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.904 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.904 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788801+0000", 2020-03-09T00:25:28.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1471000000000002e-05 2020-03-09T00:25:28.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.905 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.905 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.905 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788822+0000", 2020-03-09T00:25:28.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3599999999999998e-06 2020-03-09T00:25:28.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:28.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788826+0000", 2020-03-09T00:25:28.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066200779999999997 2020-03-09T00:25:28.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.795446+0000", 2020-03-09T00:25:28.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013750200000000001 2020-03-09T00:25:28.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.795584+0000", 2020-03-09T00:25:28.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00096939200000000002 2020-03-09T00:25:28.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796553+0000", 2020-03-09T00:25:28.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001230536 2020-03-09T00:25:28.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.797784+0000", 2020-03-09T00:25:28.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069359829999999997 2020-03-09T00:25:28.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.804720+0000", 2020-03-09T00:25:28.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010296199999999999 2020-03-09T00:25:28.913 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.914 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.914 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.804823+0000", 2020-03-09T00:25:28.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016073481000000001 2020-03-09T00:25:28.914 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.915 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.915 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.915 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.916 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:642 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:28.916 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:28.916 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.9208321860000002, 2020-03-09T00:25:28.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014044918999999999, 2020-03-09T00:25:28.916 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.917 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.917 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.917 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.917 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.918 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 642 2020-03-09T00:25:28.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.918 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:28.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:28.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0393999999999999e-05 2020-03-09T00:25:28.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785404+0000", 2020-03-09T00:25:28.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0550000000000005e-06 2020-03-09T00:25:28.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.922 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.923 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785414+0000", 2020-03-09T00:25:28.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6269999999999998e-05 2020-03-09T00:25:28.923 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785430+0000", 2020-03-09T00:25:28.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3169999999999998e-06 2020-03-09T00:25:28.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785436+0000", 2020-03-09T00:25:28.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00041141700000000002 2020-03-09T00:25:28.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785848+0000", 2020-03-09T00:25:28.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011097799999999999 2020-03-09T00:25:28.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785958+0000", 2020-03-09T00:25:28.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001069469 2020-03-09T00:25:28.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787028+0000", 2020-03-09T00:25:28.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082925840000000004 2020-03-09T00:25:28.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.795321+0000", 2020-03-09T00:25:28.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039837429999999997 2020-03-09T00:25:28.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799304+0000", 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4691999999999994e-05 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799399+0000", 2020-03-09T00:25:28.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014044918999999999 2020-03-09T00:25:28.931 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.931 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.931 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.931 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.931 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:643 3.as0 3:51058371:::smithi10428818-28:head [rollback 24] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.9206900299999998, 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014330799999999999, 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.932 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.933 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 643 2020-03-09T00:25:28.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.933 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:28.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:28.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6378999999999998e-05 2020-03-09T00:25:28.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785543+0000", 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4949999999999996e-06 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785549+0000", 2020-03-09T00:25:28.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4008e-05 2020-03-09T00:25:28.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785563+0000", 2020-03-09T00:25:28.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8459999999999999e-06 2020-03-09T00:25:28.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785568+0000", 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022776580000000001 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787846+0000", 2020-03-09T00:25:28.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011612699999999999 2020-03-09T00:25:28.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787962+0000", 2020-03-09T00:25:28.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002561486 2020-03-09T00:25:28.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.790523+0000", 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059343010000000003 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796458+0000", 2020-03-09T00:25:28.939 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003260359 2020-03-09T00:25:28.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.940 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.940 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799718+0000", 2020-03-09T00:25:28.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109141 2020-03-09T00:25:28.940 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.940 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799827+0000", 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014330799999999999 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.942 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:644 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:28.942 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:28.942 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.9205702759999999, 2020-03-09T00:25:28.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014401074, 2020-03-09T00:25:28.942 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:28.942 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:28.942 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:28.943 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:28.943 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:28.943 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 644 2020-03-09T00:25:28.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.943 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:28.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8556000000000001e-05 2020-03-09T00:25:28.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:28.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785665+0000", 2020-03-09T00:25:28.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1480000000000003e-06 2020-03-09T00:25:28.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785671+0000", 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1841999999999999e-05 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785683+0000", 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.907e-06 2020-03-09T00:25:28.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:28.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785686+0000", 2020-03-09T00:25:28.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060596110000000003 2020-03-09T00:25:28.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.791746+0000", 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013271499999999999 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.791879+0000", 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068155099999999997 2020-03-09T00:25:28.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:28.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.792560+0000", 2020-03-09T00:25:28.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039004560000000001 2020-03-09T00:25:28.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.949 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.949 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:28.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796461+0000", 2020-03-09T00:25:28.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034719059999999999 2020-03-09T00:25:28.950 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.950 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:28.950 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799933+0000", 2020-03-09T00:25:28.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4382000000000006e-05 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.800017+0000", 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014401074 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.951 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.952 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:28.952 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:28.952 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:28.954 INFO:tasks.rados.rados.0.smithi104.stdout:263: done (1 left) 2020-03-09T00:25:28.954 INFO:tasks.rados.rados.0.smithi104.stdout:264: done (0 left) 2020-03-09T00:25:28.954 INFO:tasks.rados.rados.0.smithi104.stdout:265: delete oid 17 current snap is 23 2020-03-09T00:25:28.955 INFO:tasks.rados.rados.0.smithi104.stdout:265: done (0 left) 2020-03-09T00:25:28.955 INFO:tasks.rados.rados.0.smithi104.stdout:266: delete oid 30 current snap is 23 2020-03-09T00:25:28.955 INFO:tasks.rados.rados.0.smithi104.stdout:266: done (0 left) 2020-03-09T00:25:28.955 INFO:tasks.rados.rados.0.smithi104.stdout:267: copy_from oid 37 from oid 38 current snap is 23 2020-03-09T00:25:28.955 INFO:tasks.rados.rados.0.smithi104.stdout:268: snap_create 2020-03-09T00:25:28.955 INFO:tasks.rados.rados.0.smithi104.stdout:267: finishing copy_from to smithi10428818-37 2020-03-09T00:25:28.955 INFO:tasks.rados.rados.0.smithi104.stdout:267: got expected ENOENT (src dne) 2020-03-09T00:25:28.956 INFO:tasks.rados.rados.0.smithi104.stdout:267: finishing copy_from racing read to smithi10428818-37 2020-03-09T00:25:28.956 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:28.952+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:28.957 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:25:29.055 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:29.053+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:29.085 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:29.156 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:29.157 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:29.157 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:29.159 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:29.157+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:29.159 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:25:29.259 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:29.258+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:29.275 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:29.275 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:29.275 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:29.275 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:29.276 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:29.284 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:25:29.361 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:29.359+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:29.401 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:29.402 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:29.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:29.402 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:29.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.403 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:29.403 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:29.403 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 35.547142565000001, 2020-03-09T00:25:29.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999, 2020-03-09T00:25:29.404 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.404 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.405 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.405 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:29.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.406 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.406 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.406 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:29.407 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017980700000000001 2020-03-09T00:25:29.407 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.407 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.407 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.408 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851560+0000", 2020-03-09T00:25:29.408 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5363000000000001e-05 2020-03-09T00:25:29.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.408 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.409 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851585+0000", 2020-03-09T00:25:29.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4187000000000001e-05 2020-03-09T00:25:29.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.410 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851609+0000", 2020-03-09T00:25:29.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2890000000000003e-06 2020-03-09T00:25:29.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851618+0000", 2020-03-09T00:25:29.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00036370100000000002 2020-03-09T00:25:29.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.412 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.412 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851982+0000", 2020-03-09T00:25:29.412 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3288999999999999e-05 2020-03-09T00:25:29.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851995+0000", 2020-03-09T00:25:29.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074675740000000003 2020-03-09T00:25:29.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.431 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.431 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859463+0000", 2020-03-09T00:25:29.431 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1719999999999999e-05 2020-03-09T00:25:29.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.432 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.432 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859525+0000", 2020-03-09T00:25:29.432 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999 2020-03-09T00:25:29.432 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.433 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=56, reqid=client.4751.0:254, at_version=44'23, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:25:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 35.534821557999997, 2020-03-09T00:25:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996, 2020-03-09T00:25:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015903600000000001 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863860+0000", 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3975999999999999e-05 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863884+0000", 2020-03-09T00:25:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6139999999999998e-06 2020-03-09T00:25:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.437 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863893+0000", 2020-03-09T00:25:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3190000000000002e-06 2020-03-09T00:25:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863899+0000", 2020-03-09T00:25:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039154710000000002 2020-03-09T00:25:29.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867814+0000", 2020-03-09T00:25:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5379999999999998e-05 2020-03-09T00:25:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.439 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867830+0000", 2020-03-09T00:25:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001627606 2020-03-09T00:25:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869457+0000", 2020-03-09T00:25:29.440 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7565999999999998e-05 2020-03-09T00:25:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869505+0000", 2020-03-09T00:25:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996 2020-03-09T00:25:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.441 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=77, reqid=client.4751.0:301, at_version=49'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 30.567823735000001, 2020-03-09T00:25:29.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001, 2020-03-09T00:25:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:29.443 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000394366 2020-03-09T00:25:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831093+0000", 2020-03-09T00:25:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6948999999999999e-05 2020-03-09T00:25:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831120+0000", 2020-03-09T00:25:29.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7549000000000001e-05 2020-03-09T00:25:29.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831137+0000", 2020-03-09T00:25:29.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1499999999999999e-06 2020-03-09T00:25:29.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.450 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.459 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831146+0000", 2020-03-09T00:25:29.459 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4503000000000001e-05 2020-03-09T00:25:29.459 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.459 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831230+0000", 2020-03-09T00:25:29.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4256999999999999e-05 2020-03-09T00:25:29.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831244+0000", 2020-03-09T00:25:29.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014134180999999999 2020-03-09T00:25:29.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845379+0000", 2020-03-09T00:25:29.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5120000000000002e-05 2020-03-09T00:25:29.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.463 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.463 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845424+0000", 2020-03-09T00:25:29.463 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001 2020-03-09T00:25:29.463 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.464 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.464 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.464 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.464 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=78, reqid=client.4751.0:302, at_version=49'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:29.464 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:29.464 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 30.563226581999999, 2020-03-09T00:25:29.465 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621, 2020-03-09T00:25:29.465 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.465 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.465 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.465 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.465 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:29.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:29.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166248 2020-03-09T00:25:29.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835462+0000", 2020-03-09T00:25:29.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6983000000000001e-05 2020-03-09T00:25:29.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835499+0000", 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1426000000000001e-05 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835520+0000", 2020-03-09T00:25:29.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1362000000000001e-05 2020-03-09T00:25:29.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.469 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835532+0000", 2020-03-09T00:25:29.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001298925 2020-03-09T00:25:29.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836831+0000", 2020-03-09T00:25:29.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4944e-05 2020-03-09T00:25:29.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836846+0000", 2020-03-09T00:25:29.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086294679999999995 2020-03-09T00:25:29.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.471 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.471 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845475+0000", 2020-03-09T00:25:29.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7265000000000002e-05 2020-03-09T00:25:29.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845522+0000", 2020-03-09T00:25:29.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621 2020-03-09T00:25:29.472 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.472 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.472 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.473 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=79, reqid=client.4751.0:303, at_version=49'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:29.473 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:29.474 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 30.561020893999999, 2020-03-09T00:25:29.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008, 2020-03-09T00:25:29.474 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.474 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.475 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.475 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.475 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.475 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:29.476 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.476 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:29.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0375999999999999e-05 2020-03-09T00:25:29.477 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.477 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.477 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.477 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837562+0000", 2020-03-09T00:25:29.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1155e-05 2020-03-09T00:25:29.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837583+0000", 2020-03-09T00:25:29.478 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.573e-05 2020-03-09T00:25:29.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837599+0000", 2020-03-09T00:25:29.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6429999999999998e-06 2020-03-09T00:25:29.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837605+0000", 2020-03-09T00:25:29.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074652130000000001 2020-03-09T00:25:29.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845071+0000", 2020-03-09T00:25:29.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3423e-05 2020-03-09T00:25:29.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845084+0000", 2020-03-09T00:25:29.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468407 2020-03-09T00:25:29.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845552+0000", 2020-03-09T00:25:29.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6518000000000002e-05 2020-03-09T00:25:29.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845619+0000", 2020-03-09T00:25:29.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008 2020-03-09T00:25:29.483 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.483 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.483 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.483 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/48 ECSubWrite(tid=80, reqid=client.4751.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=49'30))", 2020-03-09T00:25:29.484 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:29.484 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 30.559154164999999, 2020-03-09T00:25:29.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999, 2020-03-09T00:25:29.484 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.484 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.484 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:29.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7139999999999998e-05 2020-03-09T00:25:29.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.486 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.486 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839425+0000", 2020-03-09T00:25:29.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6959999999999999e-05 2020-03-09T00:25:29.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.486 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839452+0000", 2020-03-09T00:25:29.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2141000000000003e-05 2020-03-09T00:25:29.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839484+0000", 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2400000000000004e-06 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839491+0000", 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4830000000000002e-05 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.488 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.489 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.489 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839545+0000", 2020-03-09T00:25:29.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3210000000000001e-05 2020-03-09T00:25:29.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.489 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.489 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839559+0000", 2020-03-09T00:25:29.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0054244760000000001 2020-03-09T00:25:29.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.844983+0000", 2020-03-09T00:25:29.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9245000000000005e-05 2020-03-09T00:25:29.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845052+0000", 2020-03-09T00:25:29.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999 2020-03-09T00:25:29.491 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.491 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.491 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.492 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 53/52 ECSubWrite(tid=50, reqid=client.4751.0:311, at_version=53'13, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:29.492 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:29.492 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.610238051, 2020-03-09T00:25:29.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444, 2020-03-09T00:25:29.492 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.493 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.493 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:29.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.494 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.494 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.494 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.494 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:29.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000284932 2020-03-09T00:25:29.494 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.494 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788569+0000", 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8404e-05 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788608+0000", 2020-03-09T00:25:29.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3697e-05 2020-03-09T00:25:29.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.496 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.496 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788631+0000", 2020-03-09T00:25:29.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1178e-05 2020-03-09T00:25:29.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788642+0000", 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000149171 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788792+0000", 2020-03-09T00:25:29.497 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7829000000000002e-05 2020-03-09T00:25:29.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.498 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.498 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788809+0000", 2020-03-09T00:25:29.498 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0065850479999999996 2020-03-09T00:25:29.498 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.498 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.499 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.499 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795395+0000", 2020-03-09T00:25:29.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5185000000000001e-05 2020-03-09T00:25:29.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.499 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.499 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795450+0000", 2020-03-09T00:25:29.500 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444 2020-03-09T00:25:29.500 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.500 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.500 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.501 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:29.501 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:29.501 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.605578332, 2020-03-09T00:25:29.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999, 2020-03-09T00:25:29.501 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.501 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.502 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.502 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:29.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.503 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.503 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:29.503 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000391498 2020-03-09T00:25:29.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.503 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793335+0000", 2020-03-09T00:25:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2011999999999998e-05 2020-03-09T00:25:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793367+0000", 2020-03-09T00:25:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4936000000000001e-05 2020-03-09T00:25:29.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793382+0000", 2020-03-09T00:25:29.505 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0244000000000001e-05 2020-03-09T00:25:29.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.506 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793393+0000", 2020-03-09T00:25:29.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011101499999999999 2020-03-09T00:25:29.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.507 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.507 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793504+0000", 2020-03-09T00:25:29.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5529e-05 2020-03-09T00:25:29.507 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.507 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.508 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793519+0000", 2020-03-09T00:25:29.508 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057666640000000003 2020-03-09T00:25:29.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.508 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.508 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.508 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799286+0000", 2020-03-09T00:25:29.508 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011186100000000001 2020-03-09T00:25:29.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.509 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.509 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.509 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799398+0000", 2020-03-09T00:25:29.509 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999 2020-03-09T00:25:29.509 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.509 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.547739974999999, 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001, 2020-03-09T00:25:29.510 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.511 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.511 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:29.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.512 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.512 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:29.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001481797 2020-03-09T00:25:29.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.512 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.513 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852264+0000", 2020-03-09T00:25:29.513 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3690999999999999e-05 2020-03-09T00:25:29.513 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.513 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.513 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852308+0000", 2020-03-09T00:25:29.513 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.057e-05 2020-03-09T00:25:29.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.514 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.514 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852338+0000", 2020-03-09T00:25:29.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0482999999999999e-05 2020-03-09T00:25:29.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.515 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.515 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.515 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852349+0000", 2020-03-09T00:25:29.515 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5451999999999994e-05 2020-03-09T00:25:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.516 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852434+0000", 2020-03-09T00:25:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5016e-05 2020-03-09T00:25:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852449+0000", 2020-03-09T00:25:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022365531000000001 2020-03-09T00:25:29.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874815+0000", 2020-03-09T00:25:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7504999999999999e-05 2020-03-09T00:25:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.518 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874872+0000", 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.519 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=93, reqid=client.4751.0:380, at_version=54'42, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 25.580484746, 2020-03-09T00:25:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001, 2020-03-09T00:25:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.520 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.521 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000153535 2020-03-09T00:25:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818191+0000", 2020-03-09T00:25:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6421e-05 2020-03-09T00:25:29.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818218+0000", 2020-03-09T00:25:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7093000000000001e-05 2020-03-09T00:25:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.523 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818235+0000", 2020-03-09T00:25:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7960000000000001e-06 2020-03-09T00:25:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818243+0000", 2020-03-09T00:25:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011362 2020-03-09T00:25:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818357+0000", 2020-03-09T00:25:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2011000000000001e-05 2020-03-09T00:25:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818369+0000", 2020-03-09T00:25:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0092246010000000007 2020-03-09T00:25:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.527 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827594+0000", 2020-03-09T00:25:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1653000000000002e-05 2020-03-09T00:25:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827645+0000", 2020-03-09T00:25:29.528 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001 2020-03-09T00:25:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=94, reqid=client.4751.0:381, at_version=54'43, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 25.579189988, 2020-03-09T00:25:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005, 2020-03-09T00:25:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:29.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.531 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:29.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018645000000000001 2020-03-09T00:25:29.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.532 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.532 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819519+0000", 2020-03-09T00:25:29.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3156e-05 2020-03-09T00:25:29.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819542+0000", 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5676e-05 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819558+0000", 2020-03-09T00:25:29.533 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.742e-06 2020-03-09T00:25:29.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.534 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.534 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819564+0000", 2020-03-09T00:25:29.534 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027410770000000002 2020-03-09T00:25:29.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822305+0000", 2020-03-09T00:25:29.535 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5634999999999999e-05 2020-03-09T00:25:29.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822321+0000", 2020-03-09T00:25:29.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0053474569999999999 2020-03-09T00:25:29.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.536 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.536 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827669+0000", 2020-03-09T00:25:29.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4604999999999998e-05 2020-03-09T00:25:29.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827703+0000", 2020-03-09T00:25:29.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005 2020-03-09T00:25:29.537 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.537 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.537 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.538 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 54/32 ECSubWrite(tid=105, reqid=client.4751.0:427, at_version=54'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:29.538 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:29.538 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 25.175771434000001, 2020-03-09T00:25:29.538 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008, 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.539 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.540 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.540 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:29.540 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017923099999999999 2020-03-09T00:25:29.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.540 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.541 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222930+0000", 2020-03-09T00:25:29.541 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4340000000000001e-05 2020-03-09T00:25:29.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.541 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.541 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222954+0000", 2020-03-09T00:25:29.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4165000000000001e-05 2020-03-09T00:25:29.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222969+0000", 2020-03-09T00:25:29.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6719999999999993e-06 2020-03-09T00:25:29.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.543 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.543 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222977+0000", 2020-03-09T00:25:29.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7257999999999998e-05 2020-03-09T00:25:29.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223055+0000", 2020-03-09T00:25:29.544 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4771e-05 2020-03-09T00:25:29.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223069+0000", 2020-03-09T00:25:29.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.008721477 2020-03-09T00:25:29.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.546 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231791+0000", 2020-03-09T00:25:29.546 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3718000000000003e-05 2020-03-09T00:25:29.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.546 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.547 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231845+0000", 2020-03-09T00:25:29.547 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008 2020-03-09T00:25:29.547 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.548 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.548 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/32 ECSubWrite(tid=162, reqid=osd.11.0:162, at_version=63'46, trim_to=0'0, roll_forward_to=60'44))", 2020-03-09T00:25:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.621896927000002, 2020-03-09T00:25:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001, 2020-03-09T00:25:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.550 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.550 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.550 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:29.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.550 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.551 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.551 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:29.551 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5686999999999997e-05 2020-03-09T00:25:29.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.551 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776681+0000", 2020-03-09T00:25:29.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2730000000000001e-05 2020-03-09T00:25:29.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.552 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.552 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776704+0000", 2020-03-09T00:25:29.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4839e-05 2020-03-09T00:25:29.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.553 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.553 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776719+0000", 2020-03-09T00:25:29.553 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7049999999999999e-06 2020-03-09T00:25:29.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776726+0000", 2020-03-09T00:25:29.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00107576 2020-03-09T00:25:29.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777802+0000", 2020-03-09T00:25:29.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2907e-05 2020-03-09T00:25:29.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777815+0000", 2020-03-09T00:25:29.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0051422669999999998 2020-03-09T00:25:29.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782957+0000", 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1214e-05 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.783019+0000", 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.557 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.558 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.558 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 63/32 ECSubWrite(tid=110, reqid=client.4751.0:522, at_version=63'73, trim_to=0'0, roll_forward_to=60'71))", 2020-03-09T00:25:29.558 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:29.558 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.621408942999999, 2020-03-09T00:25:29.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001, 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.560 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:29.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7158999999999999e-05 2020-03-09T00:25:29.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.560 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777171+0000", 2020-03-09T00:25:29.561 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0669000000000001e-05 2020-03-09T00:25:29.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.561 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.561 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.561 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777191+0000", 2020-03-09T00:25:29.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5477000000000001e-05 2020-03-09T00:25:29.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777207+0000", 2020-03-09T00:25:29.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2890000000000003e-06 2020-03-09T00:25:29.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777213+0000", 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010217600000000001 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777315+0000", 2020-03-09T00:25:29.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2303999999999999e-05 2020-03-09T00:25:29.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.564 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.564 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777327+0000", 2020-03-09T00:25:29.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.005510989 2020-03-09T00:25:29.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782838+0000", 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7664000000000003e-05 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.565 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782886+0000", 2020-03-09T00:25:29.566 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001 2020-03-09T00:25:29.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.566 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.566 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.566 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 63/54 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=63'40, trim_to=0'0, roll_forward_to=60'38))", 2020-03-09T00:25:29.567 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:29.567 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.619961706000002, 2020-03-09T00:25:29.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004, 2020-03-09T00:25:29.567 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.567 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.567 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:29.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:29.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6827999999999999e-05 2020-03-09T00:25:29.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.569 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.569 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778617+0000", 2020-03-09T00:25:29.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1481999999999999e-05 2020-03-09T00:25:29.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.570 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.570 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778639+0000", 2020-03-09T00:25:29.570 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4163e-05 2020-03-09T00:25:29.570 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.570 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.571 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778653+0000", 2020-03-09T00:25:29.571 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3899999999999998e-06 2020-03-09T00:25:29.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.571 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.571 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.571 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778659+0000", 2020-03-09T00:25:29.571 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7515000000000004e-05 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778757+0000", 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2418e-05 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.572 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.573 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778769+0000", 2020-03-09T00:25:29.573 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058080179999999999 2020-03-09T00:25:29.573 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.573 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.573 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.573 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784577+0000", 2020-03-09T00:25:29.574 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3770000000000002e-05 2020-03-09T00:25:29.574 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.574 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.574 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.574 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784631+0000", 2020-03-09T00:25:29.574 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004 2020-03-09T00:25:29.574 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.575 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.575 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.575 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.575 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.575 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 63/54 ECSubWrite(tid=85, reqid=osd.14.0:85, at_version=63'28, trim_to=0'0, roll_forward_to=62'26))", 2020-03-09T00:25:29.575 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:29.576 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 16.618637903, 2020-03-09T00:25:29.576 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001, 2020-03-09T00:25:29.576 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.576 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.576 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.576 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.576 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.577 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:29.577 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.577 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.577 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.577 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.577 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:29.577 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9020000000000001e-05 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779943+0000", 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8972e-05 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.578 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.579 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779972+0000", 2020-03-09T00:25:29.579 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5282999999999999e-05 2020-03-09T00:25:29.579 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.579 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.579 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.579 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779988+0000", 2020-03-09T00:25:29.579 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5200000000000003e-06 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779994+0000", 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027147400000000001 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.580 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.581 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780266+0000", 2020-03-09T00:25:29.581 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1885000000000001e-05 2020-03-09T00:25:29.581 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.581 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.581 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.581 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780278+0000", 2020-03-09T00:25:29.581 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0050865659999999998 2020-03-09T00:25:29.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.582 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.582 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785364+0000", 2020-03-09T00:25:29.582 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6227999999999999e-05 2020-03-09T00:25:29.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.583 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.583 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785410+0000", 2020-03-09T00:25:29.583 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001 2020-03-09T00:25:29.583 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.583 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.583 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.584 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.584 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.584 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:29.584 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:29.584 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 14.596168959, 2020-03-09T00:25:29.584 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020881999999999, 2020-03-09T00:25:29.584 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.585 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.585 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.585 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.585 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.585 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:29.585 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.585 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.586 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.586 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.586 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:29.586 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0028740570000000002 2020-03-09T00:25:29.586 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.587 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.587 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.587 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805227+0000", 2020-03-09T00:25:29.587 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9300999999999999e-05 2020-03-09T00:25:29.587 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805257+0000", 2020-03-09T00:25:29.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2918000000000001e-05 2020-03-09T00:25:29.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805270+0000", 2020-03-09T00:25:29.589 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5249999999999999e-06 2020-03-09T00:25:29.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.589 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805274+0000", 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8279000000000002e-05 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805332+0000", 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3175e-05 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.590 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805346+0000", 2020-03-09T00:25:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011977272000000001 2020-03-09T00:25:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.591 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.817323+0000", 2020-03-09T00:25:29.592 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1354999999999999e-05 2020-03-09T00:25:29.592 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.592 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.592 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.592 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.817374+0000", 2020-03-09T00:25:29.592 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020881999999999 2020-03-09T00:25:29.592 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 68/67 ECSubWrite(tid=96, reqid=client.4751.0:565, at_version=68'38, trim_to=0'0, roll_forward_to=68'36))", 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.448363448, 2020-03-09T00:25:29.593 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063074259999999997, 2020-03-09T00:25:29.594 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.594 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.594 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.594 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.594 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.594 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:29.594 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.595 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.595 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.595 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.595 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:29.595 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2433000000000001e-05 2020-03-09T00:25:29.595 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.595 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950251+0000", 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2212e-05 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950274+0000", 2020-03-09T00:25:29.596 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.666e-05 2020-03-09T00:25:29.597 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.597 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.597 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.597 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950290+0000", 2020-03-09T00:25:29.597 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2519999999999996e-06 2020-03-09T00:25:29.597 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.598 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.598 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.598 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950296+0000", 2020-03-09T00:25:29.598 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8819000000000001e-05 2020-03-09T00:25:29.598 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.598 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.598 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950345+0000", 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2407e-05 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950358+0000", 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060565740000000003 2020-03-09T00:25:29.599 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.600 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.600 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.600 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956414+0000", 2020-03-09T00:25:29.600 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2068999999999999e-05 2020-03-09T00:25:29.600 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.600 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.600 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.601 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956466+0000", 2020-03-09T00:25:29.601 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063074259999999997 2020-03-09T00:25:29.601 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.601 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.601 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.601 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.601 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.602 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 68/67 ECSubWrite(tid=203, reqid=client.4751.0:575, at_version=68'66, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:29.602 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:29.602 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.445148137, 2020-03-09T00:25:29.602 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007455082, 2020-03-09T00:25:29.602 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.602 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.602 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.603 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.603 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.603 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:29.603 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.603 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.603 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.603 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.604 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:29.604 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00038396300000000002 2020-03-09T00:25:29.604 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.604 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.604 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.604 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953758+0000", 2020-03-09T00:25:29.604 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6554e-05 2020-03-09T00:25:29.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.605 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.605 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.605 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953785+0000", 2020-03-09T00:25:29.605 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7886000000000001e-05 2020-03-09T00:25:29.605 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.606 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.606 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.606 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953803+0000", 2020-03-09T00:25:29.606 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2519999999999996e-06 2020-03-09T00:25:29.606 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.606 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953809+0000", 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9489e-05 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953858+0000", 2020-03-09T00:25:29.607 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3121e-05 2020-03-09T00:25:29.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.608 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.608 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.608 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953871+0000", 2020-03-09T00:25:29.608 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068970350000000001 2020-03-09T00:25:29.608 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.608 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.609 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.609 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960768+0000", 2020-03-09T00:25:29.609 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0782000000000002e-05 2020-03-09T00:25:29.609 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.609 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.609 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.609 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960829+0000", 2020-03-09T00:25:29.610 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007455082 2020-03-09T00:25:29.610 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.610 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.610 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.610 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.610 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.611 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 68/67 ECSubWrite(tid=205, reqid=client.4751.0:577, at_version=68'68, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:29.611 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:29.611 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 11.437796109000001, 2020-03-09T00:25:29.611 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0061821239999999998, 2020-03-09T00:25:29.611 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:29.611 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:29.611 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:29.612 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:29.613 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023476109999999999 2020-03-09T00:25:29.613 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.613 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.613 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:29.613 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963074+0000", 2020-03-09T00:25:29.613 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8515000000000001e-05 2020-03-09T00:25:29.613 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.614 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.614 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:29.614 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963092+0000", 2020-03-09T00:25:29.614 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0417999999999999e-05 2020-03-09T00:25:29.614 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.614 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.614 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:29.615 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963103+0000", 2020-03-09T00:25:29.615 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8919999999999997e-06 2020-03-09T00:25:29.615 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.615 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.615 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:29.615 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963109+0000", 2020-03-09T00:25:29.615 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9573000000000002e-05 2020-03-09T00:25:29.616 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.616 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.616 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:29.616 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963148+0000", 2020-03-09T00:25:29.616 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.181e-05 2020-03-09T00:25:29.616 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.616 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.617 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:29.617 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963160+0000", 2020-03-09T00:25:29.617 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037000549999999998 2020-03-09T00:25:29.617 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.617 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.617 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.966860+0000", 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8250000000000001e-05 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.966908+0000", 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0061821239999999998 2020-03-09T00:25:29.618 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.619 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.619 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.619 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:29.619 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:29.619 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:29.621 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:25:29.621 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:29.619+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:29.721 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:29.719+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:29.758 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:29.758 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:29.758 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:29.767 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:25:29.806 INFO:tasks.rados.rados.0.smithi104.stdout:267: done (1 left) 2020-03-09T00:25:29.807 INFO:tasks.rados.rados.0.smithi104.stdout:268: done (0 left) 2020-03-09T00:25:29.807 INFO:tasks.rados.rados.0.smithi104.stdout:269: copy_from oid 25 from oid 27 current snap is 24 2020-03-09T00:25:29.807 INFO:tasks.rados.rados.0.smithi104.stdout:270: delete oid 11 current snap is 24 2020-03-09T00:25:29.814 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-03-09T00:25:29.815 INFO:tasks.rados.rados.0.smithi104.stdout:270: done (1 left) 2020-03-09T00:25:29.815 INFO:tasks.rados.rados.0.smithi104.stdout:271: read oid 31 snap -1 2020-03-09T00:25:29.815 INFO:tasks.rados.rados.0.smithi104.stdout:271: expect (ObjNum 64 snap 8 seq_num 64) 2020-03-09T00:25:29.815 INFO:tasks.rados.rados.0.smithi104.stdout:272: delete oid 24 current snap is 24 2020-03-09T00:25:29.821 INFO:tasks.rados.rados.0.smithi104.stdout:272: done (2 left) 2020-03-09T00:25:29.821 INFO:tasks.rados.rados.0.smithi104.stdout:273: delete oid 33 current snap is 24 2020-03-09T00:25:29.822 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:29.819+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:29.825 INFO:tasks.rados.rados.0.smithi104.stdout:273: done (2 left) 2020-03-09T00:25:29.825 INFO:tasks.rados.rados.0.smithi104.stdout:274: snap_remove snap 19 2020-03-09T00:25:29.900 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:29.901 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:29.901 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:29.901 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:29.901 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:29.909 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:25:29.922 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:29.921+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:30.023 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:30.021+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:30.075 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:200 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~557056] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.237527141000001, 2020-03-09T00:25:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845, 2020-03-09T00:25:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 200 2020-03-09T00:25:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001017256 2020-03-09T00:25:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835741+0000", 2020-03-09T00:25:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0451999999999999e-05 2020-03-09T00:25:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835751+0000", 2020-03-09T00:25:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1429000000000001e-05 2020-03-09T00:25:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835773+0000", 2020-03-09T00:25:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0686000000000001e-05 2020-03-09T00:25:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835783+0000", 2020-03-09T00:25:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7098e-05 2020-03-09T00:25:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835850+0000", 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100177 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835951+0000", 2020-03-09T00:25:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014653610000000001 2020-03-09T00:25:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837416+0000", 2020-03-09T00:25:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082709510000000003 2020-03-09T00:25:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845687+0000", 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015616149999999999 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847248+0000", 2020-03-09T00:25:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6819999999999998e-05 2020-03-09T00:25:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847345+0000", 2020-03-09T00:25:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845 2020-03-09T00:25:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:201 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.234818838000002, 2020-03-09T00:25:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784, 2020-03-09T00:25:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 201 2020-03-09T00:25:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081556599999999999 2020-03-09T00:25:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838247+0000", 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3170000000000004e-06 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838255+0000", 2020-03-09T00:25:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0073e-05 2020-03-09T00:25:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838265+0000", 2020-03-09T00:25:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2199999999999997e-06 2020-03-09T00:25:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838271+0000", 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003565814 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841837+0000", 2020-03-09T00:25:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013320899999999999 2020-03-09T00:25:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841970+0000", 2020-03-09T00:25:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001092275 2020-03-09T00:25:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843062+0000", 2020-03-09T00:25:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003950617 2020-03-09T00:25:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847013+0000", 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067726199999999998 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847690+0000", 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000361431 2020-03-09T00:25:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.848052+0000", 2020-03-09T00:25:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784 2020-03-09T00:25:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:202 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.233931919, 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999, 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 202 2020-03-09T00:25:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3191e-05 2020-03-09T00:25:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838362+0000", 2020-03-09T00:25:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2454e-05 2020-03-09T00:25:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838374+0000", 2020-03-09T00:25:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8269999999999998e-06 2020-03-09T00:25:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838382+0000", 2020-03-09T00:25:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1470000000000003e-06 2020-03-09T00:25:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838386+0000", 2020-03-09T00:25:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073633309999999999 2020-03-09T00:25:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845750+0000", 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103528 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845853+0000", 2020-03-09T00:25:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059542799999999995 2020-03-09T00:25:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846449+0000", 2020-03-09T00:25:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056665000000000003 2020-03-09T00:25:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847015+0000", 2020-03-09T00:25:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022036690000000001 2020-03-09T00:25:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849219+0000", 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7259e-05 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849296+0000", 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999 2020-03-09T00:25:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:203 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.233836940000003, 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296, 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 203 2020-03-09T00:25:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4102000000000001e-05 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838438+0000", 2020-03-09T00:25:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4809999999999998e-06 2020-03-09T00:25:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838441+0000", 2020-03-09T00:25:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.691e-06 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838448+0000", 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9409999999999999e-06 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838451+0000", 2020-03-09T00:25:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084379850000000003 2020-03-09T00:25:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846889+0000", 2020-03-09T00:25:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119473 2020-03-09T00:25:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847008+0000", 2020-03-09T00:25:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029378160000000002 2020-03-09T00:25:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849946+0000", 2020-03-09T00:25:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129367 2020-03-09T00:25:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.850076+0000", 2020-03-09T00:25:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013511040000000001 2020-03-09T00:25:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.851427+0000", 2020-03-09T00:25:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296 2020-03-09T00:25:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.230540482000002, 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999, 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8059000000000002e-05 2020-03-09T00:25:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841758+0000", 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3980000000000005e-06 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841768+0000", 2020-03-09T00:25:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4708e-05 2020-03-09T00:25:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841782+0000", 2020-03-09T00:25:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1330000000000001e-06 2020-03-09T00:25:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841789+0000", 2020-03-09T00:25:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037343680000000001 2020-03-09T00:25:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845524+0000", 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9321e-05 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845543+0000", 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999 2020-03-09T00:25:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.230384342000001, 2020-03-09T00:25:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002, 2020-03-09T00:25:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4193e-05 2020-03-09T00:25:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841930+0000", 2020-03-09T00:25:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0671e-05 2020-03-09T00:25:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841941+0000", 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5038000000000001e-05 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841956+0000", 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1e-06 2020-03-09T00:25:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841962+0000", 2020-03-09T00:25:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036100849999999999 2020-03-09T00:25:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845572+0000", 2020-03-09T00:25:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117707 2020-03-09T00:25:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845690+0000", 2020-03-09T00:25:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.229878661999997, 2020-03-09T00:25:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002, 2020-03-09T00:25:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6950000000000002e-05 2020-03-09T00:25:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842429+0000", 2020-03-09T00:25:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1120000000000002e-06 2020-03-09T00:25:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842438+0000", 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5486999999999999e-05 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842453+0000", 2020-03-09T00:25:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5080000000000002e-06 2020-03-09T00:25:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842460+0000", 2020-03-09T00:25:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003166791 2020-03-09T00:25:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845627+0000", 2020-03-09T00:25:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7489000000000001e-05 2020-03-09T00:25:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845644+0000", 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(osd.12.24:1 3.8s0 3:15387ad6:::smithi10428818-43:head [assert-version v5,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-03-09T00:25:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.220143616000001, 2020-03-09T00:25:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004, 2020-03-09T00:25:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0395000000000006e-05 2020-03-09T00:25:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852177+0000", 2020-03-09T00:25:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1959999999999996e-06 2020-03-09T00:25:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852184+0000", 2020-03-09T00:25:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0183000000000001e-05 2020-03-09T00:25:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852194+0000", 2020-03-09T00:25:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.463e-06 2020-03-09T00:25:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852199+0000", 2020-03-09T00:25:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7527e-05 2020-03-09T00:25:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852257+0000", 2020-03-09T00:25:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132466 2020-03-09T00:25:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852389+0000", 2020-03-09T00:25:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042799869999999999 2020-03-09T00:25:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856669+0000", 2020-03-09T00:25:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004 2020-03-09T00:25:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.20921353, 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004, 2020-03-09T00:25:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.148 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.148 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:30.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022289400000000001 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863260+0000", 2020-03-09T00:25:30.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0071000000000001e-05 2020-03-09T00:25:30.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863280+0000", 2020-03-09T00:25:30.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2445e-05 2020-03-09T00:25:30.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863292+0000", 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2320000000000001e-06 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863298+0000", 2020-03-09T00:25:30.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7059999999999998e-05 2020-03-09T00:25:30.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863365+0000", 2020-03-09T00:25:30.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6728e-05 2020-03-09T00:25:30.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863381+0000", 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048764250000000002 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868258+0000", 2020-03-09T00:25:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0857999999999999e-05 2020-03-09T00:25:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868319+0000", 2020-03-09T00:25:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004 2020-03-09T00:25:30.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 48/32 ECSubWrite(tid=43, reqid=osd.14.0:43, at_version=48'12, trim_to=0'0, roll_forward_to=44'10))", 2020-03-09T00:25:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:30.155 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.299636229000001, 2020-03-09T00:25:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001, 2020-03-09T00:25:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:30.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0922999999999999e-05 2020-03-09T00:25:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772675+0000", 2020-03-09T00:25:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8955999999999999e-05 2020-03-09T00:25:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772704+0000", 2020-03-09T00:25:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6671e-05 2020-03-09T00:25:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772721+0000", 2020-03-09T00:25:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5750999999999999e-05 2020-03-09T00:25:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772737+0000", 2020-03-09T00:25:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0823999999999998e-05 2020-03-09T00:25:30.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772807+0000", 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5262000000000001e-05 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772823+0000", 2020-03-09T00:25:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047643729999999997 2020-03-09T00:25:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777587+0000", 2020-03-09T00:25:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8967999999999993e-05 2020-03-09T00:25:30.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777666+0000", 2020-03-09T00:25:30.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001 2020-03-09T00:25:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:275 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~704512] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:30.165 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.287344496000003, 2020-03-09T00:25:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002, 2020-03-09T00:25:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 275 2020-03-09T00:25:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013228269999999999 2020-03-09T00:25:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786229+0000", 2020-03-09T00:25:30.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9299999999999997e-06 2020-03-09T00:25:30.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786236+0000", 2020-03-09T00:25:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3371000000000003e-05 2020-03-09T00:25:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786269+0000", 2020-03-09T00:25:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0840000000000003e-06 2020-03-09T00:25:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786276+0000", 2020-03-09T00:25:30.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7576e-05 2020-03-09T00:25:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786324+0000", 2020-03-09T00:25:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010013699999999999 2020-03-09T00:25:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786424+0000", 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00139653 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787821+0000", 2020-03-09T00:25:30.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089582080000000005 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796779+0000", 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036565499999999999 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797144+0000", 2020-03-09T00:25:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116684 2020-03-09T00:25:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797261+0000", 2020-03-09T00:25:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002 2020-03-09T00:25:30.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:276 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:30.179 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.285006879000001, 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001, 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 276 2020-03-09T00:25:30.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000487605 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787731+0000", 2020-03-09T00:25:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.875e-06 2020-03-09T00:25:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787739+0000", 2020-03-09T00:25:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8544000000000002e-05 2020-03-09T00:25:30.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787778+0000", 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4640999999999999e-05 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787792+0000", 2020-03-09T00:25:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038656099999999998 2020-03-09T00:25:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791658+0000", 2020-03-09T00:25:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7368999999999996e-05 2020-03-09T00:25:30.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791745+0000", 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093415700000000002 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792680+0000", 2020-03-09T00:25:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004100289 2020-03-09T00:25:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796780+0000", 2020-03-09T00:25:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074439000000000005 2020-03-09T00:25:30.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797524+0000", 2020-03-09T00:25:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2327999999999998e-05 2020-03-09T00:25:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797577+0000", 2020-03-09T00:25:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001 2020-03-09T00:25:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:277 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.284363398000004, 2020-03-09T00:25:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006, 2020-03-09T00:25:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 277 2020-03-09T00:25:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:30.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010529100000000001 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787992+0000", 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.826e-06 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787997+0000", 2020-03-09T00:25:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0430000000000008e-06 2020-03-09T00:25:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788005+0000", 2020-03-09T00:25:30.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1422e-05 2020-03-09T00:25:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788017+0000", 2020-03-09T00:25:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073510989999999998 2020-03-09T00:25:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795368+0000", 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159238 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795527+0000", 2020-03-09T00:25:30.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073327099999999997 2020-03-09T00:25:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796260+0000", 2020-03-09T00:25:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052061200000000005 2020-03-09T00:25:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796781+0000", 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095208999999999995 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797733+0000", 2020-03-09T00:25:30.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0259000000000001e-05 2020-03-09T00:25:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797783+0000", 2020-03-09T00:25:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006 2020-03-09T00:25:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.199 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:278 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-03-09T00:25:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.284170641000003, 2020-03-09T00:25:30.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999, 2020-03-09T00:25:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 278 2020-03-09T00:25:30.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3933e-05 2020-03-09T00:25:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788134+0000", 2020-03-09T00:25:30.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2150000000000007e-06 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788143+0000", 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.507e-05 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788158+0000", 2020-03-09T00:25:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.772e-05 2020-03-09T00:25:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788176+0000", 2020-03-09T00:25:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010090758 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798267+0000", 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012688700000000001 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798394+0000", 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001284739 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.799678+0000", 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999 2020-03-09T00:25:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:25:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.280197463, 2020-03-09T00:25:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004, 2020-03-09T00:25:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677000000000002e-05 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792109+0000", 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1342000000000001e-05 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792120+0000", 2020-03-09T00:25:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8834e-05 2020-03-09T00:25:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792139+0000", 2020-03-09T00:25:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1979e-05 2020-03-09T00:25:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792151+0000", 2020-03-09T00:25:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043798459999999997 2020-03-09T00:25:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796531+0000", 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.609e-05 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796547+0000", 2020-03-09T00:25:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004 2020-03-09T00:25:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:25:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.279531159000001, 2020-03-09T00:25:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002, 2020-03-09T00:25:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2404999999999996e-05 2020-03-09T00:25:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792782+0000", 2020-03-09T00:25:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2590000000000006e-06 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792791+0000", 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6633000000000002e-05 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792808+0000", 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6499999999999999e-06 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792814+0000", 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003762077 2020-03-09T00:25:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796576+0000", 2020-03-09T00:25:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8287e-05 2020-03-09T00:25:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796595+0000", 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:293 3.8s0 3:129db2f3:::smithi10428818-45:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.255663543000001, 2020-03-09T00:25:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052182940000000001, 2020-03-09T00:25:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 293 2020-03-09T00:25:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5916999999999999e-05 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816613+0000", 2020-03-09T00:25:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7960000000000007e-06 2020-03-09T00:25:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816623+0000", 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0010000000000001e-05 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816633+0000", 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0229999999999998e-06 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816639+0000", 2020-03-09T00:25:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6136e-05 2020-03-09T00:25:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816685+0000", 2020-03-09T00:25:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1557000000000005e-05 2020-03-09T00:25:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816766+0000", 2020-03-09T00:25:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095985199999999997 2020-03-09T00:25:30.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817726+0000", 2020-03-09T00:25:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037736010000000001 2020-03-09T00:25:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821500+0000", 2020-03-09T00:25:30.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021410299999999999 2020-03-09T00:25:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821714+0000", 2020-03-09T00:25:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1298999999999994e-05 2020-03-09T00:25:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821805+0000", 2020-03-09T00:25:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052182940000000001 2020-03-09T00:25:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.233 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:294 3.8s0 3:129db2f3:::smithi10428818-45:head [rollback 5] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:30.234 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.255576473000001, 2020-03-09T00:25:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060119980000000002, 2020-03-09T00:25:30.235 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 294 2020-03-09T00:25:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4272000000000001e-05 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816698+0000", 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4240000000000002e-06 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816702+0000", 2020-03-09T00:25:30.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8809999999999998e-06 2020-03-09T00:25:30.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816709+0000", 2020-03-09T00:25:30.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7520000000000001e-06 2020-03-09T00:25:30.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816711+0000", 2020-03-09T00:25:30.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019784109999999998 2020-03-09T00:25:30.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.818690+0000", 2020-03-09T00:25:30.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121984 2020-03-09T00:25:30.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.818812+0000", 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083844200000000001 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819650+0000", 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001850694 2020-03-09T00:25:30.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821501+0000", 2020-03-09T00:25:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010669200000000001 2020-03-09T00:25:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822568+0000", 2020-03-09T00:25:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118218 2020-03-09T00:25:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822686+0000", 2020-03-09T00:25:30.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060119980000000002 2020-03-09T00:25:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:295 3.8s0 3:129db2f3:::smithi10428818-45:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:30.247 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.255514064, 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067281850000000002, 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.248 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 295 2020-03-09T00:25:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5190000000000001e-05 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816762+0000", 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.061e-06 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816765+0000", 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7499999999999997e-06 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816772+0000", 2020-03-09T00:25:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.588e-06 2020-03-09T00:25:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816774+0000", 2020-03-09T00:25:30.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035648450000000001 2020-03-09T00:25:30.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.820339+0000", 2020-03-09T00:25:30.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113943 2020-03-09T00:25:30.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:30.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.820453+0000", 2020-03-09T00:25:30.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071432600000000005 2020-03-09T00:25:30.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821167+0000", 2020-03-09T00:25:30.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068605100000000002 2020-03-09T00:25:30.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821853+0000", 2020-03-09T00:25:30.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001563734 2020-03-09T00:25:30.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:30.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823417+0000", 2020-03-09T00:25:30.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7697000000000001e-05 2020-03-09T00:25:30.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823465+0000", 2020-03-09T00:25:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067281850000000002 2020-03-09T00:25:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.257 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 49/32 ECSubWriteReply(tid=27, last_complete=49'16, committed=1, applied=1))", 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.253228293999999, 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003135231, 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:30.258 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:30.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:30.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0222e-05 2020-03-09T00:25:30.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:30.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819053+0000", 2020-03-09T00:25:30.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8910000000000003e-06 2020-03-09T00:25:30.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:30.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819057+0000", 2020-03-09T00:25:30.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9330000000000001e-06 2020-03-09T00:25:30.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819065+0000", 2020-03-09T00:25:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.968e-06 2020-03-09T00:25:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819069+0000", 2020-03-09T00:25:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023769609999999999 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821446+0000", 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071125599999999995 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:30.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:30.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822158+0000", 2020-03-09T00:25:30.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003135231 2020-03-09T00:25:30.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.266 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:30.267 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:30.267 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:30.269 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:25:30.269 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:30.266+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:30.369 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:30.368+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:30.387 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:30.387 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:30.387 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:30.387 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:30.395 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:25:30.471 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:30.469+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:30.513 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:30.513 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:30.514 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:30.514 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:30.520 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:25:30.534 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:30.571 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:30.569+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:30.646 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:30.646 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:30.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:30.646 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:30.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.647 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s3 53/52 ECSubWrite(tid=50, reqid=client.4751.0:311, at_version=53'13, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:30.647 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:30.647 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 27.855092152000001, 2020-03-09T00:25:30.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013212144, 2020-03-09T00:25:30.647 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.647 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:30.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00046852800000000001 2020-03-09T00:25:30.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788727+0000", 2020-03-09T00:25:30.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6373999999999999e-05 2020-03-09T00:25:30.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788754+0000", 2020-03-09T00:25:30.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5746000000000001e-05 2020-03-09T00:25:30.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788769+0000", 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4629999999999998e-06 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788776+0000", 2020-03-09T00:25:30.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019937899999999999 2020-03-09T00:25:30.652 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.652 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.652 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.652 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788975+0000", 2020-03-09T00:25:30.652 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4996e-05 2020-03-09T00:25:30.652 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.652 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788990+0000", 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012423663999999999 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.801414+0000", 2020-03-09T00:25:30.653 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6993999999999997e-05 2020-03-09T00:25:30.654 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.654 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.654 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.654 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.801471+0000", 2020-03-09T00:25:30.654 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013212144 2020-03-09T00:25:30.654 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.654 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 27.792140118999999, 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023044938000000001, 2020-03-09T00:25:30.655 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.656 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.656 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.656 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.656 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:30.656 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059576639999999997 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857168+0000", 2020-03-09T00:25:30.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7962000000000001e-05 2020-03-09T00:25:30.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.658 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.658 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857196+0000", 2020-03-09T00:25:30.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6986999999999998e-05 2020-03-09T00:25:30.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.659 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.659 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857213+0000", 2020-03-09T00:25:30.659 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4509999999999997e-06 2020-03-09T00:25:30.659 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.659 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857220+0000", 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000106183 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857326+0000", 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.182e-05 2020-03-09T00:25:30.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.661 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.661 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.661 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857338+0000", 2020-03-09T00:25:30.661 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016865054000000001 2020-03-09T00:25:30.661 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.661 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.661 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.874203+0000", 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2816999999999999e-05 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.874256+0000", 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023044938000000001 2020-03-09T00:25:30.662 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.663 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.663 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.663 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.663 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.2s3 54/48 ECSubWrite(tid=7, reqid=client.4751.0:375, at_version=54'16, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:30.663 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:30.663 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 26.838517416999998, 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014845443, 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:30.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.665 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.665 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.665 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:30.665 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040868809999999997 2020-03-09T00:25:30.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.665 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.666 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.666 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808920+0000", 2020-03-09T00:25:30.666 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1741e-05 2020-03-09T00:25:30.666 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.666 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.666 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.666 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808952+0000", 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5818e-05 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808968+0000", 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9209999999999998e-06 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808977+0000", 2020-03-09T00:25:30.668 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018277500000000001 2020-03-09T00:25:30.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.668 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.809160+0000", 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6409999999999997e-06 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.809169+0000", 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010439004999999999 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.819608+0000", 2020-03-09T00:25:30.670 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0661000000000007e-05 2020-03-09T00:25:30.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.819679+0000", 2020-03-09T00:25:30.671 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014845443 2020-03-09T00:25:30.671 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.671 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.671 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.671 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 54/48 ECSubWrite(tid=93, reqid=client.4751.0:380, at_version=54'42, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:30.672 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.813736+0000", 2020-03-09T00:25:30.672 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 26.829615233999998, 2020-03-09T00:25:30.672 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011883421999999999, 2020-03-09T00:25:30.672 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.672 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.672 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.672 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.813736+0000", 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.813736+0000", 2020-03-09T00:25:30.673 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00063557399999999997 2020-03-09T00:25:30.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.674 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.674 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.674 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.814371+0000", 2020-03-09T00:25:30.674 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5145000000000001e-05 2020-03-09T00:25:30.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.674 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.814396+0000", 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6195e-05 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.814412+0000", 2020-03-09T00:25:30.675 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.888e-06 2020-03-09T00:25:30.676 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.676 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.814419+0000", 2020-03-09T00:25:30.676 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4152000000000005e-05 2020-03-09T00:25:30.676 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.814484+0000", 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4085e-05 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.814498+0000", 2020-03-09T00:25:30.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011065442 2020-03-09T00:25:30.678 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.678 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.678 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.678 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.825563+0000", 2020-03-09T00:25:30.678 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5940999999999999e-05 2020-03-09T00:25:30.678 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.682 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.683 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.825619+0000", 2020-03-09T00:25:30.683 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011883421999999999 2020-03-09T00:25:30.683 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.683 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.684 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.684 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.684 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:485 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:30.684 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:30.684 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.830724961000001, 2020-03-09T00:25:30.684 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028623089000000001, 2020-03-09T00:25:30.685 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.685 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.685 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.685 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.685 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.685 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 485 2020-03-09T00:25:30.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.686 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.687 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:30.687 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0157000000000001e-05 2020-03-09T00:25:30.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.687 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.687 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.687 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812676+0000", 2020-03-09T00:25:30.687 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9910000000000008e-06 2020-03-09T00:25:30.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.688 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.688 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812685+0000", 2020-03-09T00:25:30.688 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6279000000000001e-05 2020-03-09T00:25:30.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.689 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812701+0000", 2020-03-09T00:25:30.689 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7309999999999993e-06 2020-03-09T00:25:30.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.689 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812709+0000", 2020-03-09T00:25:30.690 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016644500000000001 2020-03-09T00:25:30.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.690 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.690 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.690 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812875+0000", 2020-03-09T00:25:30.690 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014119600000000001 2020-03-09T00:25:30.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.691 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.691 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.691 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813017+0000", 2020-03-09T00:25:30.691 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013810770000000001 2020-03-09T00:25:30.691 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.691 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.692 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.814398+0000", 2020-03-09T00:25:30.692 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015392095 2020-03-09T00:25:30.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.692 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829790+0000", 2020-03-09T00:25:30.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011370328000000001 2020-03-09T00:25:30.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.693 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.693 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:30.693 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841160+0000", 2020-03-09T00:25:30.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8789999999999995e-05 2020-03-09T00:25:30.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841249+0000", 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028623089000000001 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.695 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.695 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:486 3.8s0 3:1d2a75f4:::smithi10428818-31:head [rollback 14] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:30.695 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:30.695 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.830581678000001, 2020-03-09T00:25:30.695 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028685548000000002, 2020-03-09T00:25:30.695 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.695 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.696 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.696 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.696 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.696 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 486 2020-03-09T00:25:30.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.696 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.697 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:30.698 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4728999999999998e-05 2020-03-09T00:25:30.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.698 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.698 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.698 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812814+0000", 2020-03-09T00:25:30.698 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1049999999999997e-06 2020-03-09T00:25:30.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.699 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.699 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.699 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812821+0000", 2020-03-09T00:25:30.699 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2251e-05 2020-03-09T00:25:30.699 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.699 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.700 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.700 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812833+0000", 2020-03-09T00:25:30.700 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.754e-06 2020-03-09T00:25:30.700 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.700 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.700 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.700 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812840+0000", 2020-03-09T00:25:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046243279999999996 2020-03-09T00:25:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817464+0000", 2020-03-09T00:25:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121283 2020-03-09T00:25:30.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817586+0000", 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00083277700000000004 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.702 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818418+0000", 2020-03-09T00:25:30.703 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011373098999999999 2020-03-09T00:25:30.703 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.703 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.703 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.703 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829791+0000", 2020-03-09T00:25:30.703 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011588105 2020-03-09T00:25:30.704 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.704 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.704 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:30.704 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841380+0000", 2020-03-09T00:25:30.704 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5117000000000006e-05 2020-03-09T00:25:30.704 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.704 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.705 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.705 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841455+0000", 2020-03-09T00:25:30.705 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028685548000000002 2020-03-09T00:25:30.705 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.705 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.705 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.705 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.706 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.706 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:487 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:30.706 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:30.706 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.830322018, 2020-03-09T00:25:30.706 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048834447000000003, 2020-03-09T00:25:30.706 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.706 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.707 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.707 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.707 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.707 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 487 2020-03-09T00:25:30.707 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.707 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.707 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.708 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.708 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:30.708 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.708 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.708 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.708 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.708 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2626999999999997e-05 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813071+0000", 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3329999999999999e-06 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.710 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.710 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813079+0000", 2020-03-09T00:25:30.710 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5886999999999999e-05 2020-03-09T00:25:30.710 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.710 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.710 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813095+0000", 2020-03-09T00:25:30.711 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.601e-06 2020-03-09T00:25:30.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.711 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.711 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.711 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813101+0000", 2020-03-09T00:25:30.711 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020620191999999999 2020-03-09T00:25:30.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.712 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.712 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833721+0000", 2020-03-09T00:25:30.712 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013959399999999999 2020-03-09T00:25:30.712 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.713 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.713 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833861+0000", 2020-03-09T00:25:30.713 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018544080000000001 2020-03-09T00:25:30.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.713 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.713 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.713 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.835715+0000", 2020-03-09T00:25:30.714 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014671090000000001 2020-03-09T00:25:30.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.714 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.714 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.837183+0000", 2020-03-09T00:25:30.714 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024587595 2020-03-09T00:25:30.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.715 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:30.715 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861770+0000", 2020-03-09T00:25:30.715 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3101000000000002e-05 2020-03-09T00:25:30.715 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.715 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861863+0000", 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048834447000000003 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.716 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 59/54 ECSubWriteReply(tid=5, last_complete=59'26, committed=1, applied=1))", 2020-03-09T00:25:30.717 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:30.717 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.827443647999999, 2020-03-09T00:25:30.717 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013886617, 2020-03-09T00:25:30.717 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.717 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.717 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.717 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.718 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.718 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:30.718 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.718 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.718 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.718 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.718 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:30.719 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4713999999999998e-05 2020-03-09T00:25:30.719 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.719 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.719 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.719 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815962+0000", 2020-03-09T00:25:30.719 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6719999999999993e-06 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815971+0000", 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5335000000000001e-05 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.720 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.721 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815986+0000", 2020-03-09T00:25:30.721 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4869999999999999e-06 2020-03-09T00:25:30.721 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.722 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.722 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.722 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815992+0000", 2020-03-09T00:25:30.722 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013780812 2020-03-09T00:25:30.722 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829773+0000", 2020-03-09T00:25:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0596999999999999e-05 2020-03-09T00:25:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.723 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.724 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829794+0000", 2020-03-09T00:25:30.724 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013886617 2020-03-09T00:25:30.724 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.724 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.724 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.724 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.724 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 59/54 ECSubWriteReply(tid=5, last_complete=59'26, committed=1, applied=1))", 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.817605+0000", 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.825745510000001, 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012230447, 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.725 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.726 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.726 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817605+0000", 2020-03-09T00:25:30.726 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.726 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.726 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.726 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.726 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817605+0000", 2020-03-09T00:25:30.727 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3035999999999998e-05 2020-03-09T00:25:30.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.727 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.727 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.727 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817658+0000", 2020-03-09T00:25:30.727 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4789999999999996e-06 2020-03-09T00:25:30.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.728 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.728 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.728 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817667+0000", 2020-03-09T00:25:30.728 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3274e-05 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817680+0000", 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2650000000000002e-06 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.729 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.730 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817686+0000", 2020-03-09T00:25:30.730 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012137032000000001 2020-03-09T00:25:30.730 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.730 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.730 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.731 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829823+0000", 2020-03-09T00:25:30.731 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2361000000000001e-05 2020-03-09T00:25:30.731 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.731 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.731 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829836+0000", 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012230447 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 59/54 ECSubWriteReply(tid=5, last_complete=59'26, committed=1, applied=1))", 2020-03-09T00:25:30.732 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.818855+0000", 2020-03-09T00:25:30.733 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.824495518999999, 2020-03-09T00:25:30.733 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011016504999999999, 2020-03-09T00:25:30.733 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.733 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.733 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.733 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.733 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.734 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818855+0000", 2020-03-09T00:25:30.734 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.734 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.734 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.734 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.734 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818855+0000", 2020-03-09T00:25:30.734 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3952000000000001e-05 2020-03-09T00:25:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818909+0000", 2020-03-09T00:25:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1100000000000003e-06 2020-03-09T00:25:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818917+0000", 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5046e-05 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818932+0000", 2020-03-09T00:25:30.736 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1460000000000001e-06 2020-03-09T00:25:30.737 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.737 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.737 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.737 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818939+0000", 2020-03-09T00:25:30.737 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010922243999999999 2020-03-09T00:25:30.737 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.737 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.738 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.738 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829861+0000", 2020-03-09T00:25:30.738 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1007e-05 2020-03-09T00:25:30.738 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.738 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.738 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.738 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829872+0000", 2020-03-09T00:25:30.739 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011016504999999999 2020-03-09T00:25:30.739 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.739 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.739 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.739 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.739 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.739 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.799872930999999, 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017489998, 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.740 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.741 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:30.741 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.741 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.741 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.741 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.742 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:30.742 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001188824 2020-03-09T00:25:30.742 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.742 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.742 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.742 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844667+0000", 2020-03-09T00:25:30.743 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5516000000000001e-05 2020-03-09T00:25:30.743 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.743 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.743 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.743 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844692+0000", 2020-03-09T00:25:30.743 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4897e-05 2020-03-09T00:25:30.743 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844707+0000", 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3720000000000001e-06 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.744 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844713+0000", 2020-03-09T00:25:30.745 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012514399999999999 2020-03-09T00:25:30.745 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.745 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.745 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.745 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845965+0000", 2020-03-09T00:25:30.745 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3451999999999999e-05 2020-03-09T00:25:30.745 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.746 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.746 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845978+0000", 2020-03-09T00:25:30.746 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014935277 2020-03-09T00:25:30.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.746 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860914+0000", 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4219999999999999e-05 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860968+0000", 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017489998 2020-03-09T00:25:30.747 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.748 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.748 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.748 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.748 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.748 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 59/54 ECSubWrite(tid=90, reqid=client.4751.0:499, at_version=59'56, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:30.748 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.849429+0000", 2020-03-09T00:25:30.748 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.793922113000001, 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013027622000000001, 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849429+0000", 2020-03-09T00:25:30.749 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849429+0000", 2020-03-09T00:25:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000204563 2020-03-09T00:25:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.751 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.751 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849633+0000", 2020-03-09T00:25:30.751 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1787999999999999e-05 2020-03-09T00:25:30.751 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.751 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.751 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.751 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849655+0000", 2020-03-09T00:25:30.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7774e-05 2020-03-09T00:25:30.752 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.752 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.752 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.752 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849673+0000", 2020-03-09T00:25:30.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8569999999999997e-06 2020-03-09T00:25:30.753 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.753 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.753 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.753 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849680+0000", 2020-03-09T00:25:30.753 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00084582899999999998 2020-03-09T00:25:30.753 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.754 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.754 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.754 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.850526+0000", 2020-03-09T00:25:30.754 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2981e-05 2020-03-09T00:25:30.755 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.755 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.755 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.755 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.850538+0000", 2020-03-09T00:25:30.755 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011874426 2020-03-09T00:25:30.755 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.756 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.756 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.862413+0000", 2020-03-09T00:25:30.756 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3404e-05 2020-03-09T00:25:30.756 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.757 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.757 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.757 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.862456+0000", 2020-03-09T00:25:30.757 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013027622000000001 2020-03-09T00:25:30.758 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.758 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.758 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.759 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:527 3.8s0 3:1d2a75f4:::smithi10428818-31:head [rollback 13] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:30.759 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.771349+0000", 2020-03-09T00:25:30.759 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.872001595, 2020-03-09T00:25:30.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011122653999999999, 2020-03-09T00:25:30.760 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.760 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.760 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.761 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.761 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.761 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 527 2020-03-09T00:25:30.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.761 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.762 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.762 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771349+0000", 2020-03-09T00:25:30.762 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.763 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.763 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.763 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.763 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771349+0000", 2020-03-09T00:25:30.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4737000000000002e-05 2020-03-09T00:25:30.764 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.764 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.764 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.765 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771394+0000", 2020-03-09T00:25:30.765 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1580000000000002e-06 2020-03-09T00:25:30.765 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.765 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.765 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771400+0000", 2020-03-09T00:25:30.766 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1766000000000001e-05 2020-03-09T00:25:30.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.766 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771412+0000", 2020-03-09T00:25:30.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.799e-06 2020-03-09T00:25:30.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.767 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.767 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771416+0000", 2020-03-09T00:25:30.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7006000000000003e-05 2020-03-09T00:25:30.768 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.768 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.768 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771473+0000", 2020-03-09T00:25:30.768 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0516999999999999e-05 2020-03-09T00:25:30.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.769 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:30.769 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771483+0000", 2020-03-09T00:25:30.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0054910810000000001 2020-03-09T00:25:30.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776974+0000", 2020-03-09T00:25:30.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012505 2020-03-09T00:25:30.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.772 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.777099+0000", 2020-03-09T00:25:30.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00084225900000000004 2020-03-09T00:25:30.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.777942+0000", 2020-03-09T00:25:30.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0023571600000000001 2020-03-09T00:25:30.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.780299+0000", 2020-03-09T00:25:30.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00208465 2020-03-09T00:25:30.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:30.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.782383+0000", 2020-03-09T00:25:30.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8471e-05 2020-03-09T00:25:30.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.782472+0000", 2020-03-09T00:25:30.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011122653999999999 2020-03-09T00:25:30.778 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.779 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.779 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.779 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:528 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:30.780 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:30.780 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.871896956000001, 2020-03-09T00:25:30.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015882618000000001, 2020-03-09T00:25:30.780 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.780 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:30.781 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.781 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.781 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.781 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 528 2020-03-09T00:25:30.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.782 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:30.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.783 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.783 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:30.783 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5457e-05 2020-03-09T00:25:30.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771479+0000", 2020-03-09T00:25:30.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8210000000000001e-06 2020-03-09T00:25:30.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771484+0000", 2020-03-09T00:25:30.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2479999999999996e-06 2020-03-09T00:25:30.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771492+0000", 2020-03-09T00:25:30.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8449999999999999e-06 2020-03-09T00:25:30.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771495+0000", 2020-03-09T00:25:30.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4236999999999998e-05 2020-03-09T00:25:30.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771529+0000", 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5760000000000006e-06 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:30.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771538+0000", 2020-03-09T00:25:30.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078539520000000008 2020-03-09T00:25:30.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.789 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.779392+0000", 2020-03-09T00:25:30.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011161400000000001 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.779504+0000", 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00052493799999999997 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.780029+0000", 2020-03-09T00:25:30.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.006329693 2020-03-09T00:25:30.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.786358+0000", 2020-03-09T00:25:30.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00088499900000000003 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.787243+0000", 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3238000000000004e-05 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.787336+0000", 2020-03-09T00:25:30.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015882618000000001 2020-03-09T00:25:30.793 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.793 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.793 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.794 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.cs4 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:30.794 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:30.794 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 15.840941801, 2020-03-09T00:25:30.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014065318, 2020-03-09T00:25:30.795 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.795 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.795 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.795 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.795 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.795 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:30.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:30.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002574813 2020-03-09T00:25:30.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.804984+0000", 2020-03-09T00:25:30.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9797e-05 2020-03-09T00:25:30.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805004+0000", 2020-03-09T00:25:30.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1419999999999999e-06 2020-03-09T00:25:30.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805013+0000", 2020-03-09T00:25:30.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2899999999999996e-06 2020-03-09T00:25:30.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805017+0000", 2020-03-09T00:25:30.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2918999999999998e-05 2020-03-09T00:25:30.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805080+0000", 2020-03-09T00:25:30.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7710000000000001e-06 2020-03-09T00:25:30.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:30.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805090+0000", 2020-03-09T00:25:30.800 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011347336 2020-03-09T00:25:30.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.801 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:30.801 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.816437+0000", 2020-03-09T00:25:30.801 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7249999999999997e-05 2020-03-09T00:25:30.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.816474+0000", 2020-03-09T00:25:30.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014065318 2020-03-09T00:25:30.802 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.802 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.803 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.803 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:675 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~2523136,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e80)", 2020-03-09T00:25:30.803 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.814482+0000", 2020-03-09T00:25:30.804 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.82886912999999995, 2020-03-09T00:25:30.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012346322999999999, 2020-03-09T00:25:30.804 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.804 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:30.804 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.804 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.805 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.805 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 675 2020-03-09T00:25:30.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.805 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.805 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.805 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814482+0000", 2020-03-09T00:25:30.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.806 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.806 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814482+0000", 2020-03-09T00:25:30.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6347999999999995e-05 2020-03-09T00:25:30.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.807 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814558+0000", 2020-03-09T00:25:30.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0288999999999999e-05 2020-03-09T00:25:30.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814568+0000", 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6393e-05 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814585+0000", 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1040000000000001e-06 2020-03-09T00:25:30.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814592+0000", 2020-03-09T00:25:30.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8427000000000002e-05 2020-03-09T00:25:30.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814650+0000", 2020-03-09T00:25:30.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000198753 2020-03-09T00:25:30.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814849+0000", 2020-03-09T00:25:30.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011979009000000001 2020-03-09T00:25:30.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.826828+0000", 2020-03-09T00:25:30.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012346322999999999 2020-03-09T00:25:30.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.812 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.813 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:676 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~2523136,checksum] snapc 0=[] ondisk+read+known_if_redirected e80)", 2020-03-09T00:25:30.813 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:30.813 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.82869302499999997, 2020-03-09T00:25:30.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014479270000000001, 2020-03-09T00:25:30.814 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.814 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:30.814 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.814 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.814 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.815 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 676 2020-03-09T00:25:30.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.815 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.816 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:30.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:30.817 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6242000000000005e-05 2020-03-09T00:25:30.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.817 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.817 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814744+0000", 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1100000000000003e-06 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814752+0000", 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3115999999999999e-05 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814765+0000", 2020-03-09T00:25:30.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5550000000000002e-06 2020-03-09T00:25:30.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814771+0000", 2020-03-09T00:25:30.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043361559999999999 2020-03-09T00:25:30.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.819107+0000", 2020-03-09T00:25:30.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013518100000000001 2020-03-09T00:25:30.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.819242+0000", 2020-03-09T00:25:30.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098949099999999998 2020-03-09T00:25:30.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829137+0000", 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014479270000000001 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.822 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:677 3.8s0 3.2fae54b8 (undecoded) ondisk+read+known_if_redirected e80)", 2020-03-09T00:25:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.82851630899999995, 2020-03-09T00:25:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014509384, 2020-03-09T00:25:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 677 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:30.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.825 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.825 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:30.825 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6549000000000002e-05 2020-03-09T00:25:30.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.826 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.826 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814911+0000", 2020-03-09T00:25:30.826 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3329999999999997e-06 2020-03-09T00:25:30.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814919+0000", 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2802999999999999e-05 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814932+0000", 2020-03-09T00:25:30.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3369999999999996e-06 2020-03-09T00:25:30.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814939+0000", 2020-03-09T00:25:30.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014396528 2020-03-09T00:25:30.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829335+0000", 2020-03-09T00:25:30.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8340000000000005e-06 2020-03-09T00:25:30.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829344+0000", 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014509384 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.831 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:677 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~2523136,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e80)", 2020-03-09T00:25:30.831 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.828660+0000", 2020-03-09T00:25:30.831 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.81469042400000002, 2020-03-09T00:25:30.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011876769000000001, 2020-03-09T00:25:30.831 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.831 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:30.831 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.832 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.832 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.832 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 677 2020-03-09T00:25:30.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.832 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.828660+0000", 2020-03-09T00:25:30.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.828660+0000", 2020-03-09T00:25:30.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7620000000000006e-05 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.828738+0000", 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6699999999999994e-06 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.828746+0000", 2020-03-09T00:25:30.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6494000000000002e-05 2020-03-09T00:25:30.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.828762+0000", 2020-03-09T00:25:30.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4770000000000002e-06 2020-03-09T00:25:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.828768+0000", 2020-03-09T00:25:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0024574589999999999 2020-03-09T00:25:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.831225+0000", 2020-03-09T00:25:30.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014871 2020-03-09T00:25:30.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.831374+0000", 2020-03-09T00:25:30.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0091633389999999995 2020-03-09T00:25:30.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.840537+0000", 2020-03-09T00:25:30.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011876769000000001 2020-03-09T00:25:30.839 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.839 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.839 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.839 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:677 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~2523136,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e80)", 2020-03-09T00:25:30.839 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.839052+0000", 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 0.80429912000000003, 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011314090000000001, 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:30.840 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 677 2020-03-09T00:25:30.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.841 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:30.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:30.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.839052+0000", 2020-03-09T00:25:30.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:30.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.839052+0000", 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9937000000000001e-05 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:30.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.839092+0000", 2020-03-09T00:25:30.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3799999999999998e-06 2020-03-09T00:25:30.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:30.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.839095+0000", 2020-03-09T00:25:30.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8580000000000002e-06 2020-03-09T00:25:30.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.839102+0000", 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6529999999999998e-06 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:30.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.839105+0000", 2020-03-09T00:25:30.845 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035333449999999998 2020-03-09T00:25:30.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:30.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.842638+0000", 2020-03-09T00:25:30.845 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000158145 2020-03-09T00:25:30.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:30.846 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.842796+0000", 2020-03-09T00:25:30.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075697719999999998 2020-03-09T00:25:30.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:30.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:30.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:30.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.850366+0000", 2020-03-09T00:25:30.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011314090000000001 2020-03-09T00:25:30.847 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.847 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.847 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.847 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:30.848 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:30.848 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:30.851 INFO:tasks.rados.rados.0.smithi104.stdout:274: done (2 left) 2020-03-09T00:25:30.851 INFO:tasks.rados.rados.0.smithi104.stdout:275: read oid 7 snap -1 2020-03-09T00:25:30.851 INFO:tasks.rados.rados.0.smithi104.stdout:275: expect deleted 2020-03-09T00:25:30.851 INFO:tasks.rados.rados.0.smithi104.stdout:276: rmattr oid 30 current snap is 24 2020-03-09T00:25:30.851 INFO:tasks.rados.rados.0.smithi104.stdout:276: done (3 left) 2020-03-09T00:25:30.852 INFO:tasks.rados.rados.0.smithi104.stdout:277: read oid 29 snap -1 2020-03-09T00:25:30.852 INFO:tasks.rados.rados.0.smithi104.stdout:277: expect (ObjNum 80 snap 12 seq_num 80) 2020-03-09T00:25:30.852 INFO:tasks.rados.rados.0.smithi104.stdout:278: copy_from oid 16 from oid 28 current snap is 24 2020-03-09T00:25:30.852 INFO:tasks.rados.rados.0.smithi104.stdout:269: finishing copy_from to smithi10428818-25 2020-03-09T00:25:30.852 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 25 v 45 (ObjNum 78 snap 12 seq_num 78) dirty exists 2020-03-09T00:25:30.852 INFO:tasks.rados.rados.0.smithi104.stdout:269: finishing copy_from racing read to smithi10428818-25 2020-03-09T00:25:30.852 INFO:tasks.rados.rados.0.smithi104.stdout:271: expect (ObjNum 64 snap 8 seq_num 64) 2020-03-09T00:25:30.853 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:25:30.853 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:30.854 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:30.854 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:30.854 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:30.854 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:30.854 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:30.855 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:30.855 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is set 2020-03-09T00:25:30.855 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:30.847+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:30.860 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:30.881 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:30.902 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:30.920 INFO:tasks.rados.rados.0.smithi104.stdout:269: done (4 left) 2020-03-09T00:25:30.920 INFO:tasks.rados.rados.0.smithi104.stdout:271: done (3 left) 2020-03-09T00:25:30.921 INFO:tasks.rados.rados.0.smithi104.stdout:279: snap_remove snap 23 2020-03-09T00:25:30.922 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:30.942 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:30.950 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:30.949+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:30.963 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:30.983 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:30.985 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:30.997 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:25:31.005 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:31.006 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.007 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:31.007 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:31.008 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.008 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.008 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.008 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.008 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.008 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.008 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.009 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.009 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:31.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:31.051 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:31.049+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:31.127 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:31.127 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:31.128 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:25:31.152 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:31.151+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:31.253 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:31.252+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:31.257 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:31.257 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:31.266 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:31.354 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:31.353+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:31.402 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:31.402 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:31.412 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:31.455 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:31.453+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:31.537 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:31.538 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:31.545 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:31.556 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:31.555+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:31.657 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:31.655+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:31.708 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:31.708 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:31.717 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:25:31.758 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:31.756+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:31.810 INFO:tasks.rados.rados.0.smithi104.stdout:279: done (3 left) 2020-03-09T00:25:31.810 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 44 current snap is 24 2020-03-09T00:25:31.810 INFO:tasks.rados.rados.0.smithi104.stdout:280: seq_num 116 ranges {720896=786432,1507328=458752,1966080=442368,2408448=262144} 2020-03-09T00:25:31.817 INFO:tasks.rados.rados.0.smithi104.stdout:280: writing smithi10428818-44 from 720896 to 1507328 tid 1 2020-03-09T00:25:31.820 INFO:tasks.rados.rados.0.smithi104.stdout:280: writing smithi10428818-44 from 1507328 to 1966080 tid 2 2020-03-09T00:25:31.822 INFO:tasks.rados.rados.0.smithi104.stdout:280: writing smithi10428818-44 from 1966080 to 2408448 tid 3 2020-03-09T00:25:31.825 INFO:tasks.rados.rados.0.smithi104.stdout:280: writing smithi10428818-44 from 2408448 to 2670592 tid 4 2020-03-09T00:25:31.825 INFO:tasks.rados.rados.0.smithi104.stdout:281: rollback oid 26 current snap is 24 2020-03-09T00:25:31.825 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 26 to 21 2020-03-09T00:25:31.826 INFO:tasks.rados.rados.0.smithi104.stdout:282: rmattr oid 41 current snap is 24 2020-03-09T00:25:31.826 INFO:tasks.rados.rados.0.smithi104.stdout:282: done (5 left) 2020-03-09T00:25:31.826 INFO:tasks.rados.rados.0.smithi104.stdout:283: setattr oid 21 current snap is 24 2020-03-09T00:25:31.827 INFO:tasks.rados.rados.0.smithi104.stdout:277: expect (ObjNum 80 snap 12 seq_num 80) 2020-03-09T00:25:31.848 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:31.848 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:31.857 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:25:31.859 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:31.857+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:31.881 INFO:tasks.rados.rados.0.smithi104.stdout:278: finishing copy_from to smithi10428818-16 2020-03-09T00:25:31.881 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 16 v 60 (ObjNum 89 snap 14 seq_num 89) dirty exists 2020-03-09T00:25:31.882 INFO:tasks.rados.rados.0.smithi104.stdout:278: finishing copy_from racing read to smithi10428818-16 2020-03-09T00:25:31.882 INFO:tasks.rados.rados.0.smithi104.stdout:281: finishing rollback tid 0 to smithi10428818-26 2020-03-09T00:25:31.882 INFO:tasks.rados.rados.0.smithi104.stdout:281: finishing rollback tid 1 to smithi10428818-26 2020-03-09T00:25:31.882 INFO:tasks.rados.rados.0.smithi104.stdout:281: finishing rollback tid 2 to smithi10428818-26 2020-03-09T00:25:31.882 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 26 v 112 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-03-09T00:25:31.882 INFO:tasks.rados.rados.0.smithi104.stdout:280: finishing write tid 1 to smithi10428818-44 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:280: finishing write tid 2 to smithi10428818-44 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:280: finishing write tid 3 to smithi10428818-44 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:280: finishing write tid 4 to smithi10428818-44 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:280: finishing write tid 6 to smithi10428818-44 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:280: finishing write tid 7 to smithi10428818-44 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 44 v 83 (ObjNum 116 snap 24 seq_num 116) dirty exists 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:280: left oid 44 (ObjNum 116 snap 24 seq_num 116) 2020-03-09T00:25:31.883 INFO:tasks.rados.rados.0.smithi104.stdout:275: done (5 left) 2020-03-09T00:25:31.884 INFO:tasks.rados.rados.0.smithi104.stdout:277: done (4 left) 2020-03-09T00:25:31.884 INFO:tasks.rados.rados.0.smithi104.stdout:278: done (3 left) 2020-03-09T00:25:31.884 INFO:tasks.rados.rados.0.smithi104.stdout:280: done (2 left) 2020-03-09T00:25:31.884 INFO:tasks.rados.rados.0.smithi104.stdout:281: done (1 left) 2020-03-09T00:25:31.884 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 7 current snap is 24 2020-03-09T00:25:31.884 INFO:tasks.rados.rados.0.smithi104.stdout:284: seq_num 117 ranges {0=425984,425984=606208,1032192=540672,1572864=409600} 2020-03-09T00:25:31.885 INFO:tasks.rados.rados.0.smithi104.stdout:284: writing smithi10428818-7 from 0 to 425984 tid 1 2020-03-09T00:25:31.887 INFO:tasks.rados.rados.0.smithi104.stdout:284: writing smithi10428818-7 from 425984 to 1032192 tid 2 2020-03-09T00:25:31.890 INFO:tasks.rados.rados.0.smithi104.stdout:284: writing smithi10428818-7 from 1032192 to 1572864 tid 3 2020-03-09T00:25:31.893 INFO:tasks.rados.rados.0.smithi104.stdout:284: writing smithi10428818-7 from 1572864 to 1982464 tid 4 2020-03-09T00:25:31.893 INFO:tasks.rados.rados.0.smithi104.stdout:285: delete oid 6 current snap is 24 2020-03-09T00:25:31.893 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 21 v 46 (ObjNum 110 snap 21 seq_num 110) dirty exists 2020-03-09T00:25:31.894 INFO:tasks.rados.rados.0.smithi104.stdout:284: finishing write tid 1 to smithi10428818-7 2020-03-09T00:25:31.897 INFO:tasks.rados.rados.0.smithi104.stdout:283: done (2 left) 2020-03-09T00:25:31.898 INFO:tasks.rados.rados.0.smithi104.stdout:285: done (1 left) 2020-03-09T00:25:31.898 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 14 current snap is 24 2020-03-09T00:25:31.898 INFO:tasks.rados.rados.0.smithi104.stdout:286: seq_num 119 ranges {229376=458752,688128=507904,1196032=507904,1703936=458752,2162688=81920} 2020-03-09T00:25:31.901 INFO:tasks.rados.rados.0.smithi104.stdout:286: writing smithi10428818-14 from 229376 to 688128 tid 1 2020-03-09T00:25:31.908 INFO:tasks.rados.rados.0.smithi104.stdout:286: writing smithi10428818-14 from 688128 to 1196032 tid 2 2020-03-09T00:25:31.909 INFO:tasks.rados.rados.0.smithi104.stdout:286: writing smithi10428818-14 from 1196032 to 1703936 tid 3 2020-03-09T00:25:31.910 INFO:tasks.rados.rados.0.smithi104.stdout:286: writing smithi10428818-14 from 1703936 to 2162688 tid 4 2020-03-09T00:25:31.910 INFO:tasks.rados.rados.0.smithi104.stdout:286: writing smithi10428818-14 from 2162688 to 2244608 tid 5 2020-03-09T00:25:31.911 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 34 current snap is 24 2020-03-09T00:25:31.911 INFO:tasks.rados.rados.0.smithi104.stdout:287: seq_num 120 ranges {3014656=475136,3489792=655360,4145152=622592,4767744=278528} 2020-03-09T00:25:31.930 INFO:tasks.rados.rados.0.smithi104.stdout:287: writing smithi10428818-34 from 3014656 to 3489792 tid 1 2020-03-09T00:25:31.933 INFO:tasks.rados.rados.0.smithi104.stdout:287: writing smithi10428818-34 from 3489792 to 4145152 tid 2 2020-03-09T00:25:31.937 INFO:tasks.rados.rados.0.smithi104.stdout:287: writing smithi10428818-34 from 4145152 to 4767744 tid 3 2020-03-09T00:25:31.938 INFO:tasks.rados.rados.0.smithi104.stdout:287: writing smithi10428818-34 from 4767744 to 5046272 tid 4 2020-03-09T00:25:31.939 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 43 current snap is 24 2020-03-09T00:25:31.939 INFO:tasks.rados.rados.0.smithi104.stdout:288: seq_num 121 ranges {720896=491520,1212416=802816,2015232=720896,2736128=32768} 2020-03-09T00:25:31.945 INFO:tasks.rados.rados.0.smithi104.stdout:288: writing smithi10428818-43 from 720896 to 1212416 tid 1 2020-03-09T00:25:31.950 INFO:tasks.rados.rados.0.smithi104.stdout:288: writing smithi10428818-43 from 1212416 to 2015232 tid 2 2020-03-09T00:25:31.955 INFO:tasks.rados.rados.0.smithi104.stdout:288: writing smithi10428818-43 from 2015232 to 2736128 tid 3 2020-03-09T00:25:31.957 INFO:tasks.rados.rados.0.smithi104.stdout:288: writing smithi10428818-43 from 2736128 to 2768896 tid 4 2020-03-09T00:25:31.957 INFO:tasks.rados.rados.0.smithi104.stdout:289: delete oid 13 current snap is 24 2020-03-09T00:25:31.957 INFO:tasks.rados.rados.0.smithi104.stdout:284: finishing write tid 2 to smithi10428818-7 2020-03-09T00:25:31.957 INFO:tasks.rados.rados.0.smithi104.stdout:284: finishing write tid 3 to smithi10428818-7 2020-03-09T00:25:31.958 INFO:tasks.rados.rados.0.smithi104.stdout:284: finishing write tid 4 to smithi10428818-7 2020-03-09T00:25:31.958 INFO:tasks.rados.rados.0.smithi104.stdout:284: finishing write tid 6 to smithi10428818-7 2020-03-09T00:25:31.958 INFO:tasks.rados.rados.0.smithi104.stdout:284: finishing write tid 7 to smithi10428818-7 2020-03-09T00:25:31.958 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 7 v 147 (ObjNum 117 snap 24 seq_num 117) dirty exists 2020-03-09T00:25:31.959 INFO:tasks.rados.rados.0.smithi104.stdout:284: left oid 7 (ObjNum 117 snap 24 seq_num 117) 2020-03-09T00:25:31.959 INFO:tasks.rados.rados.0.smithi104.stdout:286: finishing write tid 1 to smithi10428818-14 2020-03-09T00:25:31.959 INFO:tasks.rados.rados.0.smithi104.stdout:286: finishing write tid 2 to smithi10428818-14 2020-03-09T00:25:31.959 INFO:tasks.rados.rados.0.smithi104.stdout:286: finishing write tid 3 to smithi10428818-14 2020-03-09T00:25:31.959 INFO:tasks.rados.rados.0.smithi104.stdout:286: finishing write tid 4 to smithi10428818-14 2020-03-09T00:25:31.959 INFO:tasks.rados.rados.0.smithi104.stdout:286: finishing write tid 5 to smithi10428818-14 2020-03-09T00:25:31.960 INFO:tasks.rados.rados.0.smithi104.stdout:286: finishing write tid 7 to smithi10428818-14 2020-03-09T00:25:31.960 INFO:tasks.rados.rados.0.smithi104.stdout:286: finishing write tid 8 to smithi10428818-14 2020-03-09T00:25:31.960 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 14 v 86 (ObjNum 119 snap 24 seq_num 119) dirty exists 2020-03-09T00:25:31.960 INFO:tasks.rados.rados.0.smithi104.stdout:286: left oid 14 (ObjNum 119 snap 24 seq_num 119) 2020-03-09T00:25:31.960 INFO:tasks.rados.rados.0.smithi104.stdout:287: finishing write tid 1 to smithi10428818-34 2020-03-09T00:25:31.960 INFO:tasks.rados.rados.0.smithi104.stdout:287: finishing write tid 2 to smithi10428818-34 2020-03-09T00:25:31.961 INFO:tasks.rados.rados.0.smithi104.stdout:287: finishing write tid 3 to smithi10428818-34 2020-03-09T00:25:31.962 INFO:tasks.rados.rados.0.smithi104.stdout:287: finishing write tid 4 to smithi10428818-34 2020-03-09T00:25:31.962 INFO:tasks.rados.rados.0.smithi104.stdout:287: finishing write tid 6 to smithi10428818-34 2020-03-09T00:25:31.962 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:31.960+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:31.963 INFO:tasks.rados.rados.0.smithi104.stdout:287: finishing write tid 7 to smithi10428818-34 2020-03-09T00:25:31.963 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 34 v 153 (ObjNum 120 snap 24 seq_num 120) dirty exists 2020-03-09T00:25:31.963 INFO:tasks.rados.rados.0.smithi104.stdout:287: left oid 34 (ObjNum 120 snap 24 seq_num 120) 2020-03-09T00:25:31.963 INFO:tasks.rados.rados.0.smithi104.stdout:284: done (4 left) 2020-03-09T00:25:31.963 INFO:tasks.rados.rados.0.smithi104.stdout:286: done (3 left) 2020-03-09T00:25:31.964 INFO:tasks.rados.rados.0.smithi104.stdout:287: done (2 left) 2020-03-09T00:25:31.964 INFO:tasks.rados.rados.0.smithi104.stdout:289: done (1 left) 2020-03-09T00:25:31.964 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 47 current snap is 24 2020-03-09T00:25:31.964 INFO:tasks.rados.rados.0.smithi104.stdout:290: seq_num 123 ranges {786432=524288,1310720=573440,1884160=688128,2572288=294912} 2020-03-09T00:25:31.970 INFO:tasks.rados.rados.0.smithi104.stdout:290: writing smithi10428818-47 from 786432 to 1310720 tid 1 2020-03-09T00:25:31.973 INFO:tasks.rados.rados.0.smithi104.stdout:290: writing smithi10428818-47 from 1310720 to 1884160 tid 2 2020-03-09T00:25:31.977 INFO:tasks.rados.rados.0.smithi104.stdout:290: writing smithi10428818-47 from 1884160 to 2572288 tid 3 2020-03-09T00:25:31.979 INFO:tasks.rados.rados.0.smithi104.stdout:290: writing smithi10428818-47 from 2572288 to 2867200 tid 4 2020-03-09T00:25:31.979 INFO:tasks.rados.rados.0.smithi104.stdout:291: read oid 39 snap -1 2020-03-09T00:25:31.980 INFO:tasks.rados.rados.0.smithi104.stdout:291: expect (ObjNum 89 snap 14 seq_num 89) 2020-03-09T00:25:31.980 INFO:tasks.rados.rados.0.smithi104.stdout:292: snap_remove snap 21 2020-03-09T00:25:32.004 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.005 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.014 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:25:32.062 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:32.061+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:32.131 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.132 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.139 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:25:32.163 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:32.162+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:32.255 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.255 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.263 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:25:32.265 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:32.264+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:32.366 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:32.364+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:32.378 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.379 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.387 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:25:32.467 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:32.466+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:32.542 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.542 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.550 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:25:32.568 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:32.567+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:32.669 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:32.667+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:32.682 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.682 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.691 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:25:32.769 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:32.768+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:32.815 INFO:tasks.rados.rados.0.smithi104.stdout:292: done (3 left) 2020-03-09T00:25:32.815 INFO:tasks.rados.rados.0.smithi104.stdout:293: read oid 50 snap -1 2020-03-09T00:25:32.815 INFO:tasks.rados.rados.0.smithi104.stdout:293: expect (ObjNum 91 snap 14 seq_num 91) 2020-03-09T00:25:32.816 INFO:tasks.rados.rados.0.smithi104.stdout:294: snap_remove snap 18 2020-03-09T00:25:32.849 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.849 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.856 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-03-09T00:25:32.858 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:25:32.870 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:32.868+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:32.971 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:32.969+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:32.976 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:32.976 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:32.984 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:25:33.016 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:33.072 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:33.071+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:33.121 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:33.122 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:33.131 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:25:33.148 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:33.173 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:33.172+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:33.257 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:33.257 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:33.267 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:25:33.310 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:33.309+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:33.369 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:33.375 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:33.373+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:33.389 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:33.391 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:33.391 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:33.400 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:25:33.410 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:33.431 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:33.452 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:33.473 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:33.476 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:33.475+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:33.493 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:33.514 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:33.520 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:33.520 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:33.529 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:25:33.535 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:33.557 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:33.577 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:33.577 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:33.575+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:33.598 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:33.618 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:33.640 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:33.638+0000 7f635effd700 0 -- 172.21.15.148:0/1561136054 >> [v2:172.21.15.104:6821/28043,v1:172.21.15.104:6823/28043] conn(0x7f63601865c0 msgr2=0x7f63601889b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2020-03-09T00:25:33.641 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:33.642 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.643 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.643 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.643 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.643 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.643 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.643 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:33.644 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:33.644 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.644 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.644 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.644 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.644 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.644 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.645 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.645 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:33.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:33.654 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:33.654 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:33.655 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:33.655 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:33.663 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:25:33.678 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:33.676+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:33.778 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:33.777+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:33.820 INFO:tasks.rados.rados.0.smithi104.stdout:294: done (4 left) 2020-03-09T00:25:33.821 INFO:tasks.rados.rados.0.smithi104.stdout:295: snap_remove snap 22 2020-03-09T00:25:33.821 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:33.820+0000 7efc15cb4700 -1 osd.8 pg_epoch: 84 pg[3.es0( v 83'126 (0'0,83'126] local-lis/les=32/33 n=8 ec=32/32 lis/c=32/32 les/c/f=33/33/0 sis=32) [8,9,1,11,2,0,3]p8(0) r=0 lpr=32 crt=83'126 lcod 83'124 mlcod 83'124 active+clean+snaptrim trimq=[23~1] ps=[c~1,13~1,1a~1,1f~1,22~1,24~1,26~1,2a~1,2c~1]] removing snap head 2020-03-09T00:25:33.825 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is unset 2020-03-09T00:25:33.827 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:33.827 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:33.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:33.828 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:33.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.828 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:251 3.0s0 3:0efbb6fa:::smithi10428818-44:head [copy-from ver 5] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:33.828 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:33.828 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 39.974296195999997, 2020-03-09T00:25:33.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001, 2020-03-09T00:25:33.829 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.829 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.829 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.829 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.829 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.829 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 251 2020-03-09T00:25:33.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.830 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:33.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:33.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6965999999999995e-05 2020-03-09T00:25:33.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849062+0000", 2020-03-09T00:25:33.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6689999999999995e-06 2020-03-09T00:25:33.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849071+0000", 2020-03-09T00:25:33.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7941999999999999e-05 2020-03-09T00:25:33.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849089+0000", 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9480000000000003e-06 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849099+0000", 2020-03-09T00:25:33.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4438000000000003e-05 2020-03-09T00:25:33.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849153+0000", 2020-03-09T00:25:33.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012488000000000001 2020-03-09T00:25:33.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849278+0000", 2020-03-09T00:25:33.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013412429999999999 2020-03-09T00:25:33.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862691+0000", 2020-03-09T00:25:33.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011733578 2020-03-09T00:25:33.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.874424+0000", 2020-03-09T00:25:33.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000863506 2020-03-09T00:25:33.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.840 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875288+0000", 2020-03-09T00:25:33.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9488999999999998e-05 2020-03-09T00:25:33.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875357+0000", 2020-03-09T00:25:33.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001 2020-03-09T00:25:33.841 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.841 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.841 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:252 3.0s0 3:0efbb6fa:::smithi10428818-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 39.974147397999999, 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368, 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.842 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:33.843 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.843 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.843 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.843 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 252 2020-03-09T00:25:33.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.843 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:33.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7280999999999998e-05 2020-03-09T00:25:33.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849171+0000", 2020-03-09T00:25:33.845 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6030000000000001e-06 2020-03-09T00:25:33.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849178+0000", 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.308e-05 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.846 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849191+0000", 2020-03-09T00:25:33.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.553e-06 2020-03-09T00:25:33.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.847 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849197+0000", 2020-03-09T00:25:33.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030484499999999999 2020-03-09T00:25:33.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.848 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849501+0000", 2020-03-09T00:25:33.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5923e-05 2020-03-09T00:25:33.848 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849557+0000", 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019572935999999999 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869130+0000", 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013702000000000001 2020-03-09T00:25:33.849 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.850 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.850 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:33.850 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869267+0000", 2020-03-09T00:25:33.850 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0068816229999999999 2020-03-09T00:25:33.850 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.850 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.850 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876149+0000", 2020-03-09T00:25:33.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8204999999999999e-05 2020-03-09T00:25:33.851 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876247+0000", 2020-03-09T00:25:33.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109299 2020-03-09T00:25:33.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876356+0000", 2020-03-09T00:25:33.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368 2020-03-09T00:25:33.852 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.852 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.853 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.853 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs1 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:33.853 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:33.853 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 30.972191752000001, 2020-03-09T00:25:33.853 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026627181999999999, 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.855 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.855 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:33.855 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058834270000000001 2020-03-09T00:25:33.855 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.855 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.856973+0000", 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1265999999999999e-05 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.856994+0000", 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5070000000000006e-06 2020-03-09T00:25:33.856 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.857 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.857 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.857 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857004+0000", 2020-03-09T00:25:33.857 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0560000000000004e-06 2020-03-09T00:25:33.857 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.857 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.857 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.858 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857009+0000", 2020-03-09T00:25:33.858 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1946e-05 2020-03-09T00:25:33.858 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.858 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.858 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.858 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857061+0000", 2020-03-09T00:25:33.858 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4282e-05 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857075+0000", 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020582362999999999 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.859 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.860 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.877658+0000", 2020-03-09T00:25:33.860 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9335e-05 2020-03-09T00:25:33.860 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.860 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.860 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.860 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.877717+0000", 2020-03-09T00:25:33.860 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026627181999999999 2020-03-09T00:25:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:412 3.0s0 3:054f30c0:::smithi10428818-49:head [copy-from ver 0] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:33.862 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 29.979770192, 2020-03-09T00:25:33.862 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.97896074700000002, 2020-03-09T00:25:33.862 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.862 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:33.862 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.862 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.862 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.863 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 412 2020-03-09T00:25:33.863 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.863 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.863 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.863 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.863 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:33.863 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.864 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.864 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:33.864 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014666000000000001 2020-03-09T00:25:33.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.865 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.865 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843658+0000", 2020-03-09T00:25:33.865 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0059e-05 2020-03-09T00:25:33.865 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.865 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.865 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.865 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843668+0000", 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8175000000000001e-05 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843686+0000", 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0209999999999999e-06 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.867 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.867 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843694+0000", 2020-03-09T00:25:33.867 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8402000000000001e-05 2020-03-09T00:25:33.867 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.867 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.867 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.867 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843753+0000", 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9160000000000001e-06 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.95898846199999999 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.868 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.869 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.869 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.802744+0000", 2020-03-09T00:25:33.869 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019997600000000001 2020-03-09T00:25:33.869 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.869 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.869 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.802944+0000", 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019528075999999998 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.822472+0000", 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.97896074700000002 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.870 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.871 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.871 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:413 3.0s0 3:054f30c0:::smithi10428818-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:33.871 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:33.871 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 29.979526662000001, 2020-03-09T00:25:33.871 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.98098931099999997, 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 413 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.872 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.873 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:33.873 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:33.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4046999999999999e-05 2020-03-09T00:25:33.874 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.874 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843809+0000", 2020-03-09T00:25:33.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1140000000000006e-06 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843817+0000", 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5950000000000001e-05 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.875 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.876 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843833+0000", 2020-03-09T00:25:33.876 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4799999999999998e-06 2020-03-09T00:25:33.876 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.876 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.876 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.876 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843839+0000", 2020-03-09T00:25:33.876 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6656000000000003e-05 2020-03-09T00:25:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843876+0000", 2020-03-09T00:25:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8109999999999999e-06 2020-03-09T00:25:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.878 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:33.878 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843878+0000", 2020-03-09T00:25:33.878 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.95925583699999994 2020-03-09T00:25:33.878 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.878 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.878 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.878 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803134+0000", 2020-03-09T00:25:33.879 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1947999999999997e-05 2020-03-09T00:25:33.879 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.879 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.879 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:33.879 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803196+0000", 2020-03-09T00:25:33.879 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020717148000000001 2020-03-09T00:25:33.879 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.823913+0000", 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012539800000000001 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.880 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.824038+0000", 2020-03-09T00:25:33.881 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00070592199999999999 2020-03-09T00:25:33.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.881 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.881 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.824744+0000", 2020-03-09T00:25:33.881 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.98098931099999997 2020-03-09T00:25:33.881 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.882 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.882 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.882 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.882 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:420 3.0s0 3:07071236:::smithi10428818-3:head [rollback 10] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:33.882 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:33.882 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 29.618046274000001, 2020-03-09T00:25:33.883 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61566089000000002, 2020-03-09T00:25:33.883 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.883 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.883 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.883 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.883 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.883 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 420 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.885 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.885 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:33.885 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5570000000000001e-05 2020-03-09T00:25:33.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.885 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.885 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.885 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205261+0000", 2020-03-09T00:25:33.886 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8359999999999997e-06 2020-03-09T00:25:33.886 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.886 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205267+0000", 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0376e-05 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205277+0000", 2020-03-09T00:25:33.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4229999999999999e-06 2020-03-09T00:25:33.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.888 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.888 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205284+0000", 2020-03-09T00:25:33.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2729000000000003e-05 2020-03-09T00:25:33.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.889 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.889 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205337+0000", 2020-03-09T00:25:33.889 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8240000000000001e-06 2020-03-09T00:25:33.889 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.889 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.889 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:33.889 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205341+0000", 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.59787851299999994 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803219+0000", 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000158166 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.890 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803378+0000", 2020-03-09T00:25:33.891 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012380760000000001 2020-03-09T00:25:33.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804616+0000", 2020-03-09T00:25:33.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037069310000000001 2020-03-09T00:25:33.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.808323+0000", 2020-03-09T00:25:33.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012483437999999999 2020-03-09T00:25:33.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820806+0000", 2020-03-09T00:25:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0007999999999996e-05 2020-03-09T00:25:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.894 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820896+0000", 2020-03-09T00:25:33.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61566089000000002 2020-03-09T00:25:33.894 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.894 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.894 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.895 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:421 3.0s0 3:07071236:::smithi10428818-3:head [append 0~0] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:33.896 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:33.896 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 29.617948796, 2020-03-09T00:25:33.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61577959800000004, 2020-03-09T00:25:33.897 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.897 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.897 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 421 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.899 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:33.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1856000000000001e-05 2020-03-09T00:25:33.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.899 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205364+0000", 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9029999999999997e-06 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205368+0000", 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2529999999999998e-06 2020-03-09T00:25:33.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205376+0000", 2020-03-09T00:25:33.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.004e-06 2020-03-09T00:25:33.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205379+0000", 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3745999999999998e-05 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205412+0000", 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1799999999999999e-06 2020-03-09T00:25:33.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205414+0000", 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.59956388100000002 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804977+0000", 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115327 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805093+0000", 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00070765400000000005 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805800+0000", 2020-03-09T00:25:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025236199999999999 2020-03-09T00:25:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.808324+0000", 2020-03-09T00:25:33.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012705651 2020-03-09T00:25:33.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.821030+0000", 2020-03-09T00:25:33.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2522999999999996e-05 2020-03-09T00:25:33.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.821112+0000", 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61577959800000004 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.908 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:488 3.9s0 3:93e8184b:::smithi10428818-7:head [copy-from ver 2] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:33.908 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:33.908 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.011541738999998, 2020-03-09T00:25:33.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033520886, 2020-03-09T00:25:33.908 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.908 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.909 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.909 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.909 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.909 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 488 2020-03-09T00:25:33.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.909 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.910 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:33.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.910 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.910 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7763999999999996e-05 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811817+0000", 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.093e-05 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811828+0000", 2020-03-09T00:25:33.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6980999999999999e-05 2020-03-09T00:25:33.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811845+0000", 2020-03-09T00:25:33.913 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9039999999999998e-06 2020-03-09T00:25:33.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.913 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.913 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811854+0000", 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079268800000000005 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812647+0000", 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.132e-05 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812658+0000", 2020-03-09T00:25:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043188640000000004 2020-03-09T00:25:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.816977+0000", 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109483 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817087+0000", 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011389424 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.828476+0000", 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048806689999999998 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833357+0000", 2020-03-09T00:25:33.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011728186999999999 2020-03-09T00:25:33.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845085+0000", 2020-03-09T00:25:33.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017567200000000001 2020-03-09T00:25:33.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.919 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845261+0000", 2020-03-09T00:25:33.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033520886 2020-03-09T00:25:33.919 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.919 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.919 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.920 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:489 3.9s0 3:93e8184b:::smithi10428818-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:33.920 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:33.921 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.011372967, 2020-03-09T00:25:33.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037452473999999999, 2020-03-09T00:25:33.921 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.921 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:33.921 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.922 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.922 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.922 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 489 2020-03-09T00:25:33.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2522999999999999e-05 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811951+0000", 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9319999999999999e-06 2020-03-09T00:25:33.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811958+0000", 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3498e-05 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811971+0000", 2020-03-09T00:25:33.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7749999999999998e-06 2020-03-09T00:25:33.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811977+0000", 2020-03-09T00:25:33.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00075333400000000003 2020-03-09T00:25:33.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812731+0000", 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.183e-06 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:33.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812740+0000", 2020-03-09T00:25:33.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005894598 2020-03-09T00:25:33.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818634+0000", 2020-03-09T00:25:33.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4839000000000006e-05 2020-03-09T00:25:33.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:33.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818709+0000", 2020-03-09T00:25:33.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010967121999999999 2020-03-09T00:25:33.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829676+0000", 2020-03-09T00:25:33.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013027000000000001 2020-03-09T00:25:33.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:33.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829807+0000", 2020-03-09T00:25:33.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019337825999999999 2020-03-09T00:25:33.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849144+0000", 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012293699999999999 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849267+0000", 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3636999999999998e-05 2020-03-09T00:25:33.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849361+0000", 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037452473999999999 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.934 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:490 3.0s0 3:07071236:::smithi10428818-3:head [append 0~786432] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:33.934 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:33.934 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.004315850000001, 2020-03-09T00:25:33.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049768433000000001, 2020-03-09T00:25:33.934 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.934 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.935 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.935 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.935 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.935 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 490 2020-03-09T00:25:33.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.935 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:33.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040424099999999998 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823008+0000", 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8820000000000007e-06 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823017+0000", 2020-03-09T00:25:33.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6317e-05 2020-03-09T00:25:33.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.938 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.938 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823033+0000", 2020-03-09T00:25:33.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.622e-06 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823041+0000", 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034770550000000002 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.826518+0000", 2020-03-09T00:25:33.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074775010000000001 2020-03-09T00:25:33.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833995+0000", 2020-03-09T00:25:33.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017771289999999999 2020-03-09T00:25:33.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.835772+0000", 2020-03-09T00:25:33.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0086191449999999999 2020-03-09T00:25:33.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844392+0000", 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024243164000000001 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.868635+0000", 2020-03-09T00:25:33.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9208000000000003e-05 2020-03-09T00:25:33.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.868734+0000", 2020-03-09T00:25:33.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049768433000000001 2020-03-09T00:25:33.943 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.944 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.944 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.944 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:491 3.0s0 3:07071236:::smithi10428818-3:head [append 0~491520] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:33.944 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:33.944 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 25.000180690000001, 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046647411, 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 491 2020-03-09T00:25:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.946 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:33.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00087253199999999997 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823973+0000", 2020-03-09T00:25:33.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8790000000000003e-06 2020-03-09T00:25:33.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823981+0000", 2020-03-09T00:25:33.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5770999999999999e-05 2020-03-09T00:25:33.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823997+0000", 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9669999999999998e-06 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824003+0000", 2020-03-09T00:25:33.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021062780999999999 2020-03-09T00:25:33.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.950 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845066+0000", 2020-03-09T00:25:33.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8070000000000002e-05 2020-03-09T00:25:33.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845154+0000", 2020-03-09T00:25:33.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00088644099999999997 2020-03-09T00:25:33.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.846040+0000", 2020-03-09T00:25:33.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084721699999999994 2020-03-09T00:25:33.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854512+0000", 2020-03-09T00:25:33.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015183247 2020-03-09T00:25:33.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869696+0000", 2020-03-09T00:25:33.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2553000000000003e-05 2020-03-09T00:25:33.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869748+0000", 2020-03-09T00:25:33.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046647411 2020-03-09T00:25:33.958 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.958 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.958 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.958 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:492 3.0s0 3:07071236:::smithi10428818-3:head [append 0~278528] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:33.959 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:33.959 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.999225990999999, 2020-03-09T00:25:33.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045823801999999997, 2020-03-09T00:25:33.959 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.959 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.959 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.960 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.960 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 492 2020-03-09T00:25:33.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.960 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:33.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:33.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044278900000000002 2020-03-09T00:25:33.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824498+0000", 2020-03-09T00:25:33.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7820000000000005e-06 2020-03-09T00:25:33.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824508+0000", 2020-03-09T00:25:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3557000000000001e-05 2020-03-09T00:25:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824522+0000", 2020-03-09T00:25:33.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7799999999999997e-06 2020-03-09T00:25:33.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824527+0000", 2020-03-09T00:25:33.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024667254 2020-03-09T00:25:33.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849195+0000", 2020-03-09T00:25:33.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133366 2020-03-09T00:25:33.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849328+0000", 2020-03-09T00:25:33.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001009271 2020-03-09T00:25:33.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.850337+0000", 2020-03-09T00:25:33.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016518857000000001 2020-03-09T00:25:33.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.866856+0000", 2020-03-09T00:25:33.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002975601 2020-03-09T00:25:33.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869832+0000", 2020-03-09T00:25:33.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7545e-05 2020-03-09T00:25:33.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869879+0000", 2020-03-09T00:25:33.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045823801999999997 2020-03-09T00:25:33.969 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.969 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.969 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.970 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:493 3.0s0 3:07071236:::smithi10428818-3:head [setxattr _header (56) in=7b] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:33.970 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:33.970 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.998704246999999, 2020-03-09T00:25:33.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045571699, 2020-03-09T00:25:33.970 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.971 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.971 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.971 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.971 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.971 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 493 2020-03-09T00:25:33.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.971 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:33.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:33.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1864000000000001e-05 2020-03-09T00:25:33.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824659+0000", 2020-03-09T00:25:33.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1690000000000003e-06 2020-03-09T00:25:33.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824666+0000", 2020-03-09T00:25:33.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3011e-05 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824679+0000", 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6269999999999997e-06 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824685+0000", 2020-03-09T00:25:33.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.029865282 2020-03-09T00:25:33.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854550+0000", 2020-03-09T00:25:33.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015000699999999999 2020-03-09T00:25:33.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854700+0000", 2020-03-09T00:25:33.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032691450000000002 2020-03-09T00:25:33.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:33.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.857969+0000", 2020-03-09T00:25:33.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088895759999999997 2020-03-09T00:25:33.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:33.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.866859+0000", 2020-03-09T00:25:33.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032396439999999999 2020-03-09T00:25:33.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:33.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870099+0000", 2020-03-09T00:25:33.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0374000000000003e-05 2020-03-09T00:25:33.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870149+0000", 2020-03-09T00:25:33.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045571699 2020-03-09T00:25:33.981 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.981 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.981 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.981 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:494 3.0s0 3:07071236:::smithi10428818-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:33.982 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:33.982 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.998553506, 2020-03-09T00:25:33.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048769923, 2020-03-09T00:25:33.983 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.983 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:33.984 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.984 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.984 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.984 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 494 2020-03-09T00:25:33.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.984 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:33.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1047999999999997e-05 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824769+0000", 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5699999999999998e-06 2020-03-09T00:25:33.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824776+0000", 2020-03-09T00:25:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2532e-05 2020-03-09T00:25:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824788+0000", 2020-03-09T00:25:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0990000000000001e-06 2020-03-09T00:25:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824792+0000", 2020-03-09T00:25:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035212312000000003 2020-03-09T00:25:33.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860004+0000", 2020-03-09T00:25:33.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000129932 2020-03-09T00:25:33.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:33.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860134+0000", 2020-03-09T00:25:33.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010193544000000001 2020-03-09T00:25:33.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:33.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870328+0000", 2020-03-09T00:25:33.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3980999999999995e-05 2020-03-09T00:25:33.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:33.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870402+0000", 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030959049999999999 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.873498+0000", 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048769923 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.992 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:33.993 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:33.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.993 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:506 3.0s0 3:05d2883d:::smithi10428818-38:head [stat,delete] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:33.993 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:33.993 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.989067825999999, 2020-03-09T00:25:33.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037224634, 2020-03-09T00:25:33.994 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:33.994 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:33.994 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:33.994 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:33.994 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:33.995 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 506 2020-03-09T00:25:33.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.995 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:33.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:33.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:33.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:33.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:33.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:33.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8146999999999998e-05 2020-03-09T00:25:33.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834262+0000", 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4239999999999993e-06 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:33.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834270+0000", 2020-03-09T00:25:33.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5318000000000001e-05 2020-03-09T00:25:33.999 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:33.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:33.999 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:33.999 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834285+0000", 2020-03-09T00:25:34.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5679999999999996e-06 2020-03-09T00:25:34.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834292+0000", 2020-03-09T00:25:34.000 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025873846999999998 2020-03-09T00:25:34.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860166+0000", 2020-03-09T00:25:34.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017915800000000001 2020-03-09T00:25:34.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860345+0000", 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040042910000000001 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.864349+0000", 2020-03-09T00:25:34.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029197120000000001 2020-03-09T00:25:34.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.867269+0000", 2020-03-09T00:25:34.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0041192809999999998 2020-03-09T00:25:34.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:34.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.871388+0000", 2020-03-09T00:25:34.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9888000000000001e-05 2020-03-09T00:25:34.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.871438+0000", 2020-03-09T00:25:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037224634 2020-03-09T00:25:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.006 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:530 3.9s0 3:93e8184b:::smithi10428818-7:head [append 0~475136] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:34.006 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.774914+0000", 2020-03-09T00:25:34.006 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 20.048367413000001, 2020-03-09T00:25:34.006 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025051856000000001, 2020-03-09T00:25:34.006 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:34.006 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.006 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 530 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:34.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.774914+0000", 2020-03-09T00:25:34.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:34.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:34.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.774914+0000", 2020-03-09T00:25:34.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001782779 2020-03-09T00:25:34.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:34.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776697+0000", 2020-03-09T00:25:34.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2039999999999997e-06 2020-03-09T00:25:34.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:34.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776705+0000", 2020-03-09T00:25:34.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4129e-05 2020-03-09T00:25:34.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.011 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.011 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:34.011 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776719+0000", 2020-03-09T00:25:34.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3509999999999996e-06 2020-03-09T00:25:34.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.012 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.012 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776726+0000", 2020-03-09T00:25:34.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6595999999999997e-05 2020-03-09T00:25:34.012 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776793+0000", 2020-03-09T00:25:34.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3156000000000001e-05 2020-03-09T00:25:34.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:34.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.776806+0000", 2020-03-09T00:25:34.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042757180000000004 2020-03-09T00:25:34.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781082+0000", 2020-03-09T00:25:34.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013743100000000001 2020-03-09T00:25:34.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:34.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.781219+0000", 2020-03-09T00:25:34.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015642130000000001 2020-03-09T00:25:34.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.782784+0000", 2020-03-09T00:25:34.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088951740000000005 2020-03-09T00:25:34.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.791679+0000", 2020-03-09T00:25:34.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0079903809999999995 2020-03-09T00:25:34.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:34.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.799669+0000", 2020-03-09T00:25:34.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000296724 2020-03-09T00:25:34.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:34.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.799966+0000", 2020-03-09T00:25:34.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025051856000000001 2020-03-09T00:25:34.019 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:535 3.9s0 3:93e8184b:::smithi10428818-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e64)", 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 20.039544759000002, 2020-03-09T00:25:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026643350999999999, 2020-03-09T00:25:34.021 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:34.021 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:34.021 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:34.021 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:34.021 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.021 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 535 2020-03-09T00:25:34.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.022 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:34.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:34.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:34.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:34.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5562999999999997e-05 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783782+0000", 2020-03-09T00:25:34.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5560000000000004e-06 2020-03-09T00:25:34.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:34.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783789+0000", 2020-03-09T00:25:34.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2455e-05 2020-03-09T00:25:34.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:34.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783801+0000", 2020-03-09T00:25:34.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.386e-06 2020-03-09T00:25:34.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783806+0000", 2020-03-09T00:25:34.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018586522000000001 2020-03-09T00:25:34.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.802392+0000", 2020-03-09T00:25:34.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7225999999999995e-05 2020-03-09T00:25:34.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:34.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.802479+0000", 2020-03-09T00:25:34.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051016660000000004 2020-03-09T00:25:34.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.807581+0000", 2020-03-09T00:25:34.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6997999999999998e-05 2020-03-09T00:25:34.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:34.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.807648+0000", 2020-03-09T00:25:34.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027319789999999998 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.810380+0000", 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026643350999999999 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:34.029 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.030 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:543 3.9s0 3:9f11c47d:::smithi10428818-37:head [append 0~0] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:34.030 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.784011+0000", 2020-03-09T00:25:34.030 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 20.039270426000002, 2020-03-09T00:25:34.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025124014, 2020-03-09T00:25:34.030 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 543 2020-03-09T00:25:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.031 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784011+0000", 2020-03-09T00:25:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784011+0000", 2020-03-09T00:25:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0240000000000001e-05 2020-03-09T00:25:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784051+0000", 2020-03-09T00:25:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3740000000000003e-06 2020-03-09T00:25:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784058+0000", 2020-03-09T00:25:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.152e-05 2020-03-09T00:25:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784069+0000", 2020-03-09T00:25:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6880000000000002e-06 2020-03-09T00:25:34.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.784076+0000", 2020-03-09T00:25:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020748419000000001 2020-03-09T00:25:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.804824+0000", 2020-03-09T00:25:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115017 2020-03-09T00:25:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.804939+0000", 2020-03-09T00:25:34.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00071181800000000002 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.805651+0000", 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00090687999999999999 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.806558+0000", 2020-03-09T00:25:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025351779999999999 2020-03-09T00:25:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.809093+0000", 2020-03-09T00:25:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1879999999999999e-05 2020-03-09T00:25:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.809135+0000", 2020-03-09T00:25:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025124014 2020-03-09T00:25:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.040 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:687 3.9s0 3:91ab8712:::smithi10428818-16:head [copy-from ver 60] snapc 2c=[2c,2b,2a,23] ondisk+write+known_if_redirected e81)", 2020-03-09T00:25:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:30.805722+0000", 2020-03-09T00:25:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.0175594179999998, 2020-03-09T00:25:34.041 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038719972999999998, 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 687 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.042 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805722+0000", 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:34.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805722+0000", 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2537e-05 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805795+0000", 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8480000000000003e-06 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805801+0000", 2020-03-09T00:25:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5707999999999999e-05 2020-03-09T00:25:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:34.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805817+0000", 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9030000000000003e-06 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805822+0000", 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018435500000000001 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806006+0000", 2020-03-09T00:25:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0726e-05 2020-03-09T00:25:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:34.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806017+0000", 2020-03-09T00:25:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020965160000000001 2020-03-09T00:25:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808114+0000", 2020-03-09T00:25:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119157 2020-03-09T00:25:34.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808233+0000", 2020-03-09T00:25:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020385461000000001 2020-03-09T00:25:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.828618+0000", 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010904479 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.839523+0000", 2020-03-09T00:25:34.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047344329999999997 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.844257+0000", 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018484999999999999 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.844442+0000", 2020-03-09T00:25:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038719972999999998 2020-03-09T00:25:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:688 3.9s0 3:91ab8712:::smithi10428818-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e81)", 2020-03-09T00:25:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:30.805872+0000", 2020-03-09T00:25:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.0174091679999999, 2020-03-09T00:25:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039786291000000001, 2020-03-09T00:25:34.053 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 688 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:34.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805872+0000", 2020-03-09T00:25:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:34.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805872+0000", 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2169000000000001e-05 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805924+0000", 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3419999999999996e-06 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805931+0000", 2020-03-09T00:25:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.322e-05 2020-03-09T00:25:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:34.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805944+0000", 2020-03-09T00:25:34.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8330000000000001e-06 2020-03-09T00:25:34.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.058 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805949+0000", 2020-03-09T00:25:34.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6842000000000006e-05 2020-03-09T00:25:34.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806046+0000", 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5959999999999993e-06 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:34.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806055+0000", 2020-03-09T00:25:34.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002393306 2020-03-09T00:25:34.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808449+0000", 2020-03-09T00:25:34.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0771e-05 2020-03-09T00:25:34.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:34.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808489+0000", 2020-03-09T00:25:34.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.029870654999999999 2020-03-09T00:25:34.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.838360+0000", 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015941600000000001 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.838519+0000", 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0068704389999999999 2020-03-09T00:25:34.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:34.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.845390+0000", 2020-03-09T00:25:34.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7422000000000003e-05 2020-03-09T00:25:34.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:34.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.845487+0000", 2020-03-09T00:25:34.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017128 2020-03-09T00:25:34.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:34.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:34.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:34.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.845659+0000", 2020-03-09T00:25:34.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039786291000000001 2020-03-09T00:25:34.065 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.065 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:34.065 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.065 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:34.065 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:34.065 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:34.067 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:25:34.068 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:34.065+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:34.167 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:34.166+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:34.185 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:34.186 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:34.186 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:34.194 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:25:34.268 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:34.266+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:34.309 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:34.309 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:34.310 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:34.310 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:34.317 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:25:34.369 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:34.367+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:34.470 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:34.468+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:34.477 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:196 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~540672] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:34.477 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.648541532000003, 2020-03-09T00:25:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501, 2020-03-09T00:25:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 196 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.479 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035415000000000002 2020-03-09T00:25:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.480 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826046+0000", 2020-03-09T00:25:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2260000000000005e-06 2020-03-09T00:25:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826054+0000", 2020-03-09T00:25:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3873000000000001e-05 2020-03-09T00:25:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826068+0000", 2020-03-09T00:25:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2020-03-09T00:25:34.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826074+0000", 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0126e-05 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826134+0000", 2020-03-09T00:25:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001117 2020-03-09T00:25:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826246+0000", 2020-03-09T00:25:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00124165 2020-03-09T00:25:34.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827488+0000", 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088469059999999999 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836335+0000", 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035733999999999999 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836692+0000", 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1440000000000005e-05 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836783+0000", 2020-03-09T00:25:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501 2020-03-09T00:25:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:197 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.643282995, 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097135529999999998, 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.488 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 197 2020-03-09T00:25:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.830950+0000", 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027437700000000002 2020-03-09T00:25:34.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831225+0000", 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8239999999999996e-06 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831231+0000", 2020-03-09T00:25:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3345e-05 2020-03-09T00:25:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831244+0000", 2020-03-09T00:25:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4499999999999998e-06 2020-03-09T00:25:34.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831251+0000", 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00057306900000000005 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831824+0000", 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103781 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831928+0000", 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000811518 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832740+0000", 2020-03-09T00:25:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036368210000000002 2020-03-09T00:25:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836377+0000", 2020-03-09T00:25:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0042093749999999996 2020-03-09T00:25:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840586+0000", 2020-03-09T00:25:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7992999999999997e-05 2020-03-09T00:25:34.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840664+0000", 2020-03-09T00:25:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097135529999999998 2020-03-09T00:25:34.497 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:198 3.5s0 3:ada90060:::smithi10428818-8:head [setxattr _header (55) in=7b] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:34.498 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.642927311000001, 2020-03-09T00:25:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095424670000000007, 2020-03-09T00:25:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.499 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 198 2020-03-09T00:25:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:34.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831306+0000", 2020-03-09T00:25:34.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9859999999999999e-05 2020-03-09T00:25:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831376+0000", 2020-03-09T00:25:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3319999999999997e-06 2020-03-09T00:25:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831382+0000", 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1367e-05 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831394+0000", 2020-03-09T00:25:34.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4580000000000003e-06 2020-03-09T00:25:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831398+0000", 2020-03-09T00:25:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039429479999999999 2020-03-09T00:25:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835341+0000", 2020-03-09T00:25:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011724 2020-03-09T00:25:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.835458+0000", 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068325099999999995 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836141+0000", 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00092122099999999995 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.837063+0000", 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036999060000000002 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840763+0000", 2020-03-09T00:25:34.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5884000000000006e-05 2020-03-09T00:25:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.840848+0000", 2020-03-09T00:25:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095424670000000007 2020-03-09T00:25:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.508 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:199 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.642795413000002, 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149, 2020-03-09T00:25:34.509 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 199 2020-03-09T00:25:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.510 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.511 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1242999999999998e-05 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831479+0000", 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5740000000000001e-06 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831486+0000", 2020-03-09T00:25:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1513e-05 2020-03-09T00:25:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831497+0000", 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2760000000000002e-06 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831502+0000", 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053448499999999999 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836846+0000", 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103844 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836950+0000", 2020-03-09T00:25:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045940299999999998 2020-03-09T00:25:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841544+0000", 2020-03-09T00:25:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105018 2020-03-09T00:25:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.516 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841649+0000", 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002376801 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.844026+0000", 2020-03-09T00:25:34.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149 2020-03-09T00:25:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:346 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:34.518 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.648701815999999, 2020-03-09T00:25:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654, 2020-03-09T00:25:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.519 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 346 2020-03-09T00:25:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:34.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014902900000000001 2020-03-09T00:25:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825681+0000", 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825687+0000", 2020-03-09T00:25:34.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6099999999999995e-06 2020-03-09T00:25:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825696+0000", 2020-03-09T00:25:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6149999999999999e-06 2020-03-09T00:25:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825701+0000", 2020-03-09T00:25:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4975000000000002e-05 2020-03-09T00:25:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825746+0000", 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0597e-05 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825836+0000", 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091597100000000002 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826752+0000", 2020-03-09T00:25:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046291539999999999 2020-03-09T00:25:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831382+0000", 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098416760000000006 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841223+0000", 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100926 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841324+0000", 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:347 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~622592] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.641433994, 2020-03-09T00:25:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999, 2020-03-09T00:25:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.529 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 347 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000469071 2020-03-09T00:25:34.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833268+0000", 2020-03-09T00:25:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7889999999999995e-06 2020-03-09T00:25:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833276+0000", 2020-03-09T00:25:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6033999999999999e-05 2020-03-09T00:25:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833292+0000", 2020-03-09T00:25:34.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8850000000000001e-06 2020-03-09T00:25:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833299+0000", 2020-03-09T00:25:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833343+0000", 2020-03-09T00:25:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105574 2020-03-09T00:25:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833448+0000", 2020-03-09T00:25:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094406100000000005 2020-03-09T00:25:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834392+0000", 2020-03-09T00:25:34.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088904150000000005 2020-03-09T00:25:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843283+0000", 2020-03-09T00:25:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052783500000000004 2020-03-09T00:25:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843811+0000", 2020-03-09T00:25:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1547999999999998e-05 2020-03-09T00:25:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843902+0000", 2020-03-09T00:25:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999 2020-03-09T00:25:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:348 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~147456] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.640877904, 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999, 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.540 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 348 2020-03-09T00:25:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019096900000000001 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833546+0000", 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4060000000000002e-06 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833553+0000", 2020-03-09T00:25:34.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1321000000000001e-05 2020-03-09T00:25:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6310000000000002e-06 2020-03-09T00:25:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833569+0000", 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037440260000000001 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837313+0000", 2020-03-09T00:25:34.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134988 2020-03-09T00:25:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837448+0000", 2020-03-09T00:25:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001155454 2020-03-09T00:25:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.838603+0000", 2020-03-09T00:25:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046829050000000002 2020-03-09T00:25:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843286+0000", 2020-03-09T00:25:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011927330000000001 2020-03-09T00:25:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844479+0000", 2020-03-09T00:25:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5680000000000005e-05 2020-03-09T00:25:34.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844575+0000", 2020-03-09T00:25:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999 2020-03-09T00:25:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.549 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:349 3.5s0 3:ada90060:::smithi10428818-8:head [setxattr _header (56) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.640612344000001, 2020-03-09T00:25:34.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001, 2020-03-09T00:25:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 349 2020-03-09T00:25:34.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8177999999999997e-05 2020-03-09T00:25:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833689+0000", 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3619999999999997e-06 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833695+0000", 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1154e-05 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833706+0000", 2020-03-09T00:25:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5030000000000002e-06 2020-03-09T00:25:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833712+0000", 2020-03-09T00:25:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0079814919999999998 2020-03-09T00:25:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841694+0000", 2020-03-09T00:25:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013282600000000001 2020-03-09T00:25:34.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841826+0000", 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071321600000000002 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.842540+0000", 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074733700000000004 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843287+0000", 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001845368 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845132+0000", 2020-03-09T00:25:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000004e-05 2020-03-09T00:25:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845219+0000", 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:350 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:34.560 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 31.640482686999999, 2020-03-09T00:25:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417, 2020-03-09T00:25:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.561 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 350 2020-03-09T00:25:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:34.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0074999999999998e-05 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833791+0000", 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1530000000000003e-06 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833800+0000", 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3621000000000001e-05 2020-03-09T00:25:34.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833813+0000", 2020-03-09T00:25:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9899999999999999e-06 2020-03-09T00:25:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833817+0000", 2020-03-09T00:25:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0093306459999999997 2020-03-09T00:25:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843148+0000", 2020-03-09T00:25:34.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012790100000000001 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843276+0000", 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025486789999999999 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845825+0000", 2020-03-09T00:25:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120695 2020-03-09T00:25:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845945+0000", 2020-03-09T00:25:34.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033866569999999999 2020-03-09T00:25:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.849332+0000", 2020-03-09T00:25:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417 2020-03-09T00:25:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.569 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.8s5 59/54 ECSubWrite(tid=6, reqid=client.4751.0:486, at_version=59'27, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.655341884999999, 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012816759, 2020-03-09T00:25:34.570 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102045 2020-03-09T00:25:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818993+0000", 2020-03-09T00:25:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2311999999999999e-05 2020-03-09T00:25:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819016+0000", 2020-03-09T00:25:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6019999999999999e-05 2020-03-09T00:25:34.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819032+0000", 2020-03-09T00:25:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4910000000000002e-06 2020-03-09T00:25:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819038+0000", 2020-03-09T00:25:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023069140000000002 2020-03-09T00:25:34.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821345+0000", 2020-03-09T00:25:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3864e-05 2020-03-09T00:25:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821359+0000", 2020-03-09T00:25:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010305182 2020-03-09T00:25:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831664+0000", 2020-03-09T00:25:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3930999999999997e-05 2020-03-09T00:25:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831708+0000", 2020-03-09T00:25:34.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012816759 2020-03-09T00:25:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.0s2 59/54 ECSubWrite(tid=159, reqid=osd.12.0:159, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:34.582 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.821409+0000", 2020-03-09T00:25:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.652824532, 2020-03-09T00:25:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010250832, 2020-03-09T00:25:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.583 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821409+0000", 2020-03-09T00:25:34.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821409+0000", 2020-03-09T00:25:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8247e-05 2020-03-09T00:25:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821467+0000", 2020-03-09T00:25:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4675e-05 2020-03-09T00:25:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821492+0000", 2020-03-09T00:25:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5407e-05 2020-03-09T00:25:34.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821507+0000", 2020-03-09T00:25:34.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1920000000000002e-06 2020-03-09T00:25:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821513+0000", 2020-03-09T00:25:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102107 2020-03-09T00:25:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821615+0000", 2020-03-09T00:25:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3492999999999999e-05 2020-03-09T00:25:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821629+0000", 2020-03-09T00:25:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0099815000000000008 2020-03-09T00:25:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831610+0000", 2020-03-09T00:25:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9211e-05 2020-03-09T00:25:34.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831660+0000", 2020-03-09T00:25:34.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010250832 2020-03-09T00:25:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s4 59/48 ECSubWrite(tid=23, reqid=osd.9.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=59'30))", 2020-03-09T00:25:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.822203+0000", 2020-03-09T00:25:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.652030123999999, 2020-03-09T00:25:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.009406078, 2020-03-09T00:25:34.597 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822203+0000", 2020-03-09T00:25:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822203+0000", 2020-03-09T00:25:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8763e-05 2020-03-09T00:25:34.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822262+0000", 2020-03-09T00:25:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4282e-05 2020-03-09T00:25:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822276+0000", 2020-03-09T00:25:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5724000000000001e-05 2020-03-09T00:25:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822292+0000", 2020-03-09T00:25:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2359999999999998e-06 2020-03-09T00:25:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822299+0000", 2020-03-09T00:25:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000229817 2020-03-09T00:25:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822529+0000", 2020-03-09T00:25:34.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3582e-05 2020-03-09T00:25:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.822543+0000", 2020-03-09T00:25:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.009010611 2020-03-09T00:25:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831553+0000", 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6063000000000002e-05 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831609+0000", 2020-03-09T00:25:34.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.009406078 2020-03-09T00:25:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 59/32 ECSubWrite(tid=162, reqid=osd.12.0:162, at_version=59'79, trim_to=0'0, roll_forward_to=58'75))", 2020-03-09T00:25:34.606 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.648635927000001, 2020-03-09T00:25:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013484734, 2020-03-09T00:25:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120926 2020-03-09T00:25:34.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825718+0000", 2020-03-09T00:25:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8563000000000003e-05 2020-03-09T00:25:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825757+0000", 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3974999999999999e-05 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825791+0000", 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2510999999999999e-05 2020-03-09T00:25:34.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825803+0000", 2020-03-09T00:25:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062712599999999999 2020-03-09T00:25:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826430+0000", 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2381000000000001e-05 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826443+0000", 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012584521 2020-03-09T00:25:34.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839027+0000", 2020-03-09T00:25:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4731000000000002e-05 2020-03-09T00:25:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839082+0000", 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013484734 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.614 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 59/32 ECSubWrite(tid=163, reqid=client.4751.0:488, at_version=59'80, trim_to=0'0, roll_forward_to=58'75))", 2020-03-09T00:25:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.829002+0000", 2020-03-09T00:25:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.645231414000001, 2020-03-09T00:25:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010120809, 2020-03-09T00:25:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829002+0000", 2020-03-09T00:25:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829002+0000", 2020-03-09T00:25:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018660200000000001 2020-03-09T00:25:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829189+0000", 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3999999999999999e-05 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829233+0000", 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2617000000000002e-05 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829275+0000", 2020-03-09T00:25:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1619999999999996e-06 2020-03-09T00:25:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.829281+0000", 2020-03-09T00:25:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002744841 2020-03-09T00:25:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.832026+0000", 2020-03-09T00:25:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2108e-05 2020-03-09T00:25:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.832038+0000", 2020-03-09T00:25:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0070503889999999998 2020-03-09T00:25:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839089+0000", 2020-03-09T00:25:34.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4090000000000001e-05 2020-03-09T00:25:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839123+0000", 2020-03-09T00:25:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010120809 2020-03-09T00:25:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(osd.11.24:6 3.5s0 3:a0e51e4d:::smithi10428818-41:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e75)", 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.784676+0000", 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 9.6895571050000004, 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010000617, 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784676+0000", 2020-03-09T00:25:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784676+0000", 2020-03-09T00:25:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8823000000000002e-05 2020-03-09T00:25:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784745+0000", 2020-03-09T00:25:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3530000000000002e-06 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784750+0000", 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0118e-05 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784761+0000", 2020-03-09T00:25:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9599999999999997e-06 2020-03-09T00:25:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784766+0000", 2020-03-09T00:25:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3801999999999997e-05 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784830+0000", 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2255999999999999e-05 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.784843+0000", 2020-03-09T00:25:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0031329869999999998 2020-03-09T00:25:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787976+0000", 2020-03-09T00:25:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012125000000000001 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.788097+0000", 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0065800679999999997 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.794677+0000", 2020-03-09T00:25:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010000617 2020-03-09T00:25:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:650 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~3244032,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e76)", 2020-03-09T00:25:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:25.784887+0000", 2020-03-09T00:25:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.6893461359999993, 2020-03-09T00:25:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017635736999999999, 2020-03-09T00:25:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 650 2020-03-09T00:25:34.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784887+0000", 2020-03-09T00:25:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784887+0000", 2020-03-09T00:25:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5030999999999995e-05 2020-03-09T00:25:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784952+0000", 2020-03-09T00:25:34.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4569999999999999e-06 2020-03-09T00:25:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784960+0000", 2020-03-09T00:25:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4148000000000001e-05 2020-03-09T00:25:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784974+0000", 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2119999999999997e-06 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784981+0000", 2020-03-09T00:25:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4878e-05 2020-03-09T00:25:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785026+0000", 2020-03-09T00:25:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5020000000000003e-06 2020-03-09T00:25:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785033+0000", 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002150463 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787184+0000", 2020-03-09T00:25:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000112349 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787296+0000", 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015226697000000001 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.802523+0000", 2020-03-09T00:25:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017635736999999999 2020-03-09T00:25:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:651 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~3244032,checksum] snapc 0=[] ondisk+read+known_if_redirected e76)", 2020-03-09T00:25:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:25.785035+0000", 2020-03-09T00:25:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.6891982609999996, 2020-03-09T00:25:34.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020421043999999999, 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 651 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.651 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785035+0000", 2020-03-09T00:25:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785035+0000", 2020-03-09T00:25:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5442000000000001e-05 2020-03-09T00:25:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785101+0000", 2020-03-09T00:25:34.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5069999999999997e-06 2020-03-09T00:25:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785107+0000", 2020-03-09T00:25:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0242e-05 2020-03-09T00:25:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785117+0000", 2020-03-09T00:25:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7919999999999999e-06 2020-03-09T00:25:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785121+0000", 2020-03-09T00:25:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9161000000000001e-05 2020-03-09T00:25:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785150+0000", 2020-03-09T00:25:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6180000000000003e-06 2020-03-09T00:25:34.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785156+0000", 2020-03-09T00:25:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025586839999999999 2020-03-09T00:25:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787715+0000", 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011526799999999999 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787830+0000", 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017626329999999999 2020-03-09T00:25:34.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.805456+0000", 2020-03-09T00:25:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020421043999999999 2020-03-09T00:25:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.659 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:652 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~3244032,checksum] snapc 0=[] ondisk+read+known_if_redirected e76)", 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:25.785168+0000", 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.6890654600000001, 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023422449000000001, 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.660 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 652 2020-03-09T00:25:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785168+0000", 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785168+0000", 2020-03-09T00:25:34.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1877000000000004e-05 2020-03-09T00:25:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785230+0000", 2020-03-09T00:25:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0100000000000003e-06 2020-03-09T00:25:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785235+0000", 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5959999999999993e-06 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785244+0000", 2020-03-09T00:25:34.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8240000000000001e-06 2020-03-09T00:25:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785248+0000", 2020-03-09T00:25:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1299e-05 2020-03-09T00:25:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785279+0000", 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6649999999999998e-06 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785289+0000", 2020-03-09T00:25:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029630780000000001 2020-03-09T00:25:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.788252+0000", 2020-03-09T00:25:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012021199999999999 2020-03-09T00:25:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.788372+0000", 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020217888 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.808590+0000", 2020-03-09T00:25:34.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023422449000000001 2020-03-09T00:25:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s3 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:34.669 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:29.820910+0000", 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 4.6533230430000003, 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010875378, 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.820910+0000", 2020-03-09T00:25:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.820910+0000", 2020-03-09T00:25:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00049114299999999998 2020-03-09T00:25:34.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821401+0000", 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7106e-05 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821439+0000", 2020-03-09T00:25:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6792000000000001e-05 2020-03-09T00:25:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821455+0000", 2020-03-09T00:25:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8630000000000008e-06 2020-03-09T00:25:34.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821463+0000", 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000380322 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821844+0000", 2020-03-09T00:25:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7122e-05 2020-03-09T00:25:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821861+0000", 2020-03-09T00:25:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098754129999999996 2020-03-09T00:25:34.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.831736+0000", 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9617000000000003e-05 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.831786+0000", 2020-03-09T00:25:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010875378 2020-03-09T00:25:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.677 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 81/67 ECSubWrite(tid=263, reqid=client.4751.0:687, at_version=81'142, trim_to=0'0, roll_forward_to=71'139))", 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:30.829030+0000", 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.6452033880000001, 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01118852, 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.829030+0000", 2020-03-09T00:25:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.829030+0000", 2020-03-09T00:25:34.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013312039999999999 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830361+0000", 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2396e-05 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830384+0000", 2020-03-09T00:25:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1953999999999999e-05 2020-03-09T00:25:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830395+0000", 2020-03-09T00:25:34.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4450000000000004e-06 2020-03-09T00:25:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830401+0000", 2020-03-09T00:25:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.371e-05 2020-03-09T00:25:34.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830445+0000", 2020-03-09T00:25:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0094e-05 2020-03-09T00:25:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.683 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830455+0000", 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097253349999999999 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.840180+0000", 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8381999999999999e-05 2020-03-09T00:25:34.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.840218+0000", 2020-03-09T00:25:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01118852 2020-03-09T00:25:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:34.686 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:34.687 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:34.686+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:34.688 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:25:34.788 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:34.786+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:34.815 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:34.815 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:34.815 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:34.816 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:34.822 INFO:tasks.rados.rados.0.smithi104.stdout:295: done (4 left) 2020-03-09T00:25:34.823 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 22 current snap is 24 2020-03-09T00:25:34.823 INFO:tasks.rados.rados.0.smithi104.stdout:296: seq_num 124 ranges {2310144=540672,2850816=720896,3571712=786432,4358144=49152} 2020-03-09T00:25:34.823 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:34.822+0000 7f853d4c9700 -1 osd.9 pg_epoch: 85 pg[3.2s0( v 84'62 (0'0,84'62] local-lis/les=48/49 n=3 ec=32/32 lis/c=48/48 les/c/f=49/49/0 sis=48) [9,12,3,4,15,1,11]p9(0) r=0 lpr=48 crt=84'62 lcod 83'60 mlcod 83'60 active+clean+snaptrim trimq=[2b~1] ps=[c~1,13~1,1a~1,1f~1,22~3,26~1,2a~1,2c~1]] removing snap head 2020-03-09T00:25:34.825 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:25:34.838 INFO:tasks.rados.rados.0.smithi104.stdout:296: writing smithi10428818-22 from 2310144 to 2850816 tid 1 2020-03-09T00:25:34.842 INFO:tasks.rados.rados.0.smithi104.stdout:296: writing smithi10428818-22 from 2850816 to 3571712 tid 2 2020-03-09T00:25:34.847 INFO:tasks.rados.rados.0.smithi104.stdout:296: writing smithi10428818-22 from 3571712 to 4358144 tid 3 2020-03-09T00:25:34.847 INFO:tasks.rados.rados.0.smithi104.stdout:296: writing smithi10428818-22 from 4358144 to 4407296 tid 4 2020-03-09T00:25:34.847 INFO:tasks.rados.rados.0.smithi104.stdout:297: rollback oid 3 current snap is 24 2020-03-09T00:25:34.848 INFO:tasks.rados.rados.0.smithi104.stdout:297: done (5 left) 2020-03-09T00:25:34.848 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 31 current snap is 24 2020-03-09T00:25:34.848 INFO:tasks.rados.rados.0.smithi104.stdout:298: seq_num 125 ranges {2523136=557056,3080192=475136,3555328=507904,4063232=475136,4538368=98304} 2020-03-09T00:25:34.864 INFO:tasks.rados.rados.0.smithi104.stdout:298: writing smithi10428818-31 from 2523136 to 3080192 tid 1 2020-03-09T00:25:34.866 INFO:tasks.rados.rados.0.smithi104.stdout:298: writing smithi10428818-31 from 3080192 to 3555328 tid 2 2020-03-09T00:25:34.869 INFO:tasks.rados.rados.0.smithi104.stdout:298: writing smithi10428818-31 from 3555328 to 4063232 tid 3 2020-03-09T00:25:34.872 INFO:tasks.rados.rados.0.smithi104.stdout:298: writing smithi10428818-31 from 4063232 to 4538368 tid 4 2020-03-09T00:25:34.873 INFO:tasks.rados.rados.0.smithi104.stdout:298: writing smithi10428818-31 from 4538368 to 4636672 tid 5 2020-03-09T00:25:34.873 INFO:tasks.rados.rados.0.smithi104.stdout:299: read oid 41 snap -1 2020-03-09T00:25:34.873 INFO:tasks.rados.rados.0.smithi104.stdout:299: expect (ObjNum 40 snap 0 seq_num 40) 2020-03-09T00:25:34.874 INFO:tasks.rados.rados.0.smithi104.stdout:300: copy_from oid 19 from oid 12 current snap is 24 2020-03-09T00:25:34.874 INFO:tasks.rados.rados.0.smithi104.stdout:288: finishing write tid 1 to smithi10428818-43 2020-03-09T00:25:34.874 INFO:tasks.rados.rados.0.smithi104.stdout:288: finishing write tid 2 to smithi10428818-43 2020-03-09T00:25:34.874 INFO:tasks.rados.rados.0.smithi104.stdout:288: finishing write tid 3 to smithi10428818-43 2020-03-09T00:25:34.874 INFO:tasks.rados.rados.0.smithi104.stdout:288: finishing write tid 4 to smithi10428818-43 2020-03-09T00:25:34.874 INFO:tasks.rados.rados.0.smithi104.stdout:288: finishing write tid 6 to smithi10428818-43 2020-03-09T00:25:34.874 INFO:tasks.rados.rados.0.smithi104.stdout:288: finishing write tid 7 to smithi10428818-43 2020-03-09T00:25:34.875 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 43 v 46 (ObjNum 121 snap 24 seq_num 121) dirty exists 2020-03-09T00:25:34.875 INFO:tasks.rados.rados.0.smithi104.stdout:288: left oid 43 (ObjNum 121 snap 24 seq_num 121) 2020-03-09T00:25:34.875 INFO:tasks.rados.rados.0.smithi104.stdout:290: finishing write tid 1 to smithi10428818-47 2020-03-09T00:25:34.875 INFO:tasks.rados.rados.0.smithi104.stdout:288: done (7 left) 2020-03-09T00:25:34.875 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 10 current snap is 24 2020-03-09T00:25:34.875 INFO:tasks.rados.rados.0.smithi104.stdout:301: seq_num 126 ranges {1048576=573440,1622016=622592,2244608=622592,2867200=311296} 2020-03-09T00:25:34.882 INFO:tasks.rados.rados.0.smithi104.stdout:301: writing smithi10428818-10 from 1048576 to 1622016 tid 1 2020-03-09T00:25:34.886 INFO:tasks.rados.rados.0.smithi104.stdout:301: writing smithi10428818-10 from 1622016 to 2244608 tid 2 2020-03-09T00:25:34.890 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:34.887+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:34.891 INFO:tasks.rados.rados.0.smithi104.stdout:301: writing smithi10428818-10 from 2244608 to 2867200 tid 3 2020-03-09T00:25:34.893 INFO:tasks.rados.rados.0.smithi104.stdout:301: writing smithi10428818-10 from 2867200 to 3178496 tid 4 2020-03-09T00:25:34.894 INFO:tasks.rados.rados.0.smithi104.stdout:302: read oid 45 snap -1 2020-03-09T00:25:34.894 INFO:tasks.rados.rados.0.smithi104.stdout:302: expect (ObjNum 44 snap 0 seq_num 44) 2020-03-09T00:25:34.894 INFO:tasks.rados.rados.0.smithi104.stdout:303: rollback oid 24 current snap is 24 2020-03-09T00:25:34.894 INFO:tasks.rados.rados.0.smithi104.stdout:303: done (9 left) 2020-03-09T00:25:34.894 INFO:tasks.rados.rados.0.smithi104.stdout:304: delete oid 23 current snap is 24 2020-03-09T00:25:34.894 INFO:tasks.rados.rados.0.smithi104.stdout:291: expect (ObjNum 89 snap 14 seq_num 89) 2020-03-09T00:25:34.905 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:34.898+0000 7efc11cac700 -1 osd.8 pg_epoch: 85 pg[3.es0( v 85'136 (0'0,85'136] local-lis/les=32/33 n=5 ec=32/32 lis/c=32/32 les/c/f=33/33/0 sis=32) [8,9,1,11,2,0,3]p8(0) r=0 lpr=32 luod=84'134 crt=84'134 lcod 84'132 mlcod 84'132 active+clean+snaptrim trimq=[2b~1] ps=[c~1,13~1,1a~1,1f~1,22~3,26~1,2a~1,2c~1]] removing snap head 2020-03-09T00:25:34.956 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:34.957 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:34.957 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:34.957 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:34.957 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:34.960 INFO:tasks.rados.rados.0.smithi104.stdout:290: finishing write tid 2 to smithi10428818-47 2020-03-09T00:25:34.960 INFO:tasks.rados.rados.0.smithi104.stdout:290: finishing write tid 3 to smithi10428818-47 2020-03-09T00:25:34.960 INFO:tasks.rados.rados.0.smithi104.stdout:290: finishing write tid 4 to smithi10428818-47 2020-03-09T00:25:34.960 INFO:tasks.rados.rados.0.smithi104.stdout:290: finishing write tid 6 to smithi10428818-47 2020-03-09T00:25:34.960 INFO:tasks.rados.rados.0.smithi104.stdout:290: finishing write tid 7 to smithi10428818-47 2020-03-09T00:25:34.960 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 47 v 120 (ObjNum 123 snap 24 seq_num 123) dirty exists 2020-03-09T00:25:34.961 INFO:tasks.rados.rados.0.smithi104.stdout:290: left oid 47 (ObjNum 123 snap 24 seq_num 123) 2020-03-09T00:25:34.961 INFO:tasks.rados.rados.0.smithi104.stdout:293: expect (ObjNum 91 snap 14 seq_num 91) 2020-03-09T00:25:34.965 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:25:34.989 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:34.988+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:35.028 INFO:tasks.rados.rados.0.smithi104.stdout:296: finishing write tid 1 to smithi10428818-22 2020-03-09T00:25:35.028 INFO:tasks.rados.rados.0.smithi104.stdout:296: finishing write tid 2 to smithi10428818-22 2020-03-09T00:25:35.029 INFO:tasks.rados.rados.0.smithi104.stdout:296: finishing write tid 3 to smithi10428818-22 2020-03-09T00:25:35.029 INFO:tasks.rados.rados.0.smithi104.stdout:296: finishing write tid 4 to smithi10428818-22 2020-03-09T00:25:35.029 INFO:tasks.rados.rados.0.smithi104.stdout:296: finishing write tid 6 to smithi10428818-22 2020-03-09T00:25:35.029 INFO:tasks.rados.rados.0.smithi104.stdout:296: finishing write tid 7 to smithi10428818-22 2020-03-09T00:25:35.029 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 22 v 158 (ObjNum 124 snap 24 seq_num 124) dirty exists 2020-03-09T00:25:35.030 INFO:tasks.rados.rados.0.smithi104.stdout:296: left oid 22 (ObjNum 124 snap 24 seq_num 124) 2020-03-09T00:25:35.030 INFO:tasks.rados.rados.0.smithi104.stdout:290: done (9 left) 2020-03-09T00:25:35.030 INFO:tasks.rados.rados.0.smithi104.stdout:291: done (8 left) 2020-03-09T00:25:35.030 INFO:tasks.rados.rados.0.smithi104.stdout:293: done (7 left) 2020-03-09T00:25:35.030 INFO:tasks.rados.rados.0.smithi104.stdout:296: done (6 left) 2020-03-09T00:25:35.030 INFO:tasks.rados.rados.0.smithi104.stdout:304: done (5 left) 2020-03-09T00:25:35.031 INFO:tasks.rados.rados.0.smithi104.stdout:305: delete oid 47 current snap is 24 2020-03-09T00:25:35.031 INFO:tasks.rados.rados.0.smithi104.stdout:298: finishing write tid 1 to smithi10428818-31 2020-03-09T00:25:35.031 INFO:tasks.rados.rados.0.smithi104.stdout:299: expect (ObjNum 40 snap 0 seq_num 40) 2020-03-09T00:25:35.046 INFO:tasks.rados.rados.0.smithi104.stdout:298: finishing write tid 2 to smithi10428818-31 2020-03-09T00:25:35.046 INFO:tasks.rados.rados.0.smithi104.stdout:298: finishing write tid 3 to smithi10428818-31 2020-03-09T00:25:35.046 INFO:tasks.rados.rados.0.smithi104.stdout:298: finishing write tid 4 to smithi10428818-31 2020-03-09T00:25:35.046 INFO:tasks.rados.rados.0.smithi104.stdout:298: finishing write tid 5 to smithi10428818-31 2020-03-09T00:25:35.047 INFO:tasks.rados.rados.0.smithi104.stdout:298: finishing write tid 7 to smithi10428818-31 2020-03-09T00:25:35.047 INFO:tasks.rados.rados.0.smithi104.stdout:298: finishing write tid 8 to smithi10428818-31 2020-03-09T00:25:35.047 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 31 v 52 (ObjNum 125 snap 24 seq_num 125) dirty exists 2020-03-09T00:25:35.047 INFO:tasks.rados.rados.0.smithi104.stdout:298: left oid 31 (ObjNum 125 snap 24 seq_num 125) 2020-03-09T00:25:35.047 INFO:tasks.rados.rados.0.smithi104.stdout:301: finishing write tid 1 to smithi10428818-10 2020-03-09T00:25:35.047 INFO:tasks.rados.rados.0.smithi104.stdout:298: done (5 left) 2020-03-09T00:25:35.047 INFO:tasks.rados.rados.0.smithi104.stdout:299: done (4 left) 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:305: done (3 left) 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:306: rollback oid 33 current snap is 24 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:306: done (3 left) 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:307: snap_create 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:301: finishing write tid 2 to smithi10428818-10 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:300: finishing copy_from to smithi10428818-19 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 19 v 24 (ObjNum 75 snap 11 seq_num 75) dirty exists 2020-03-09T00:25:35.048 INFO:tasks.rados.rados.0.smithi104.stdout:302: expect (ObjNum 44 snap 0 seq_num 44) 2020-03-09T00:25:35.064 INFO:tasks.rados.rados.0.smithi104.stdout:301: finishing write tid 3 to smithi10428818-10 2020-03-09T00:25:35.064 INFO:tasks.rados.rados.0.smithi104.stdout:300: finishing copy_from racing read to smithi10428818-19 2020-03-09T00:25:35.064 INFO:tasks.rados.rados.0.smithi104.stdout:301: finishing write tid 4 to smithi10428818-10 2020-03-09T00:25:35.064 INFO:tasks.rados.rados.0.smithi104.stdout:301: finishing write tid 6 to smithi10428818-10 2020-03-09T00:25:35.064 INFO:tasks.rados.rados.0.smithi104.stdout:301: finishing write tid 7 to smithi10428818-10 2020-03-09T00:25:35.065 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 10 v 51 (ObjNum 126 snap 24 seq_num 126) dirty exists 2020-03-09T00:25:35.065 INFO:tasks.rados.rados.0.smithi104.stdout:301: left oid 10 (ObjNum 126 snap 24 seq_num 126) 2020-03-09T00:25:35.083 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:35.083 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:35.083 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:35.084 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:35.084 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.084 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:311 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~770048] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:35.084 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:35.084 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.296895345000003, 2020-03-09T00:25:35.084 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438, 2020-03-09T00:25:35.084 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.085 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.085 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.085 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 311 2020-03-09T00:25:35.092 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.092 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.092 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.092 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.092 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:35.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.093 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.093 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.093 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:35.093 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014001199999999999 2020-03-09T00:25:35.093 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.094 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785132+0000", 2020-03-09T00:25:35.094 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1973999999999999e-05 2020-03-09T00:25:35.094 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.094 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.094 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785144+0000", 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1815e-05 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785156+0000", 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2449999999999998e-06 2020-03-09T00:25:35.095 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785161+0000", 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00063616300000000003 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.096 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785797+0000", 2020-03-09T00:25:35.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00065134999999999998 2020-03-09T00:25:35.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.097 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.097 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.097 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.786449+0000", 2020-03-09T00:25:35.097 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00188737 2020-03-09T00:25:35.097 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788336+0000", 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010887625999999999 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.098 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799224+0000", 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037060560000000001 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.802930+0000", 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000460719 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.099 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803390+0000", 2020-03-09T00:25:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438 2020-03-09T00:25:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.100 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:312 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~376832] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.295399197000002, 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001, 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.101 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 312 2020-03-09T00:25:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.102 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026595500000000001 2020-03-09T00:25:35.103 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785494+0000", 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3902e-05 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.104 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785508+0000", 2020-03-09T00:25:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.417e-05 2020-03-09T00:25:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785522+0000", 2020-03-09T00:25:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.99e-06 2020-03-09T00:25:35.105 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785529+0000", 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083854199999999993 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.106 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793915+0000", 2020-03-09T00:25:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024809000000000002 2020-03-09T00:25:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.794163+0000", 2020-03-09T00:25:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001122184 2020-03-09T00:25:35.107 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795285+0000", 2020-03-09T00:25:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004034358 2020-03-09T00:25:35.108 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799319+0000", 2020-03-09T00:25:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042626369999999997 2020-03-09T00:25:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.109 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803582+0000", 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00042245199999999997 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804004+0000", 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001 2020-03-09T00:25:35.110 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:313 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.295043636000003, 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001, 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 313 2020-03-09T00:25:35.112 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7169000000000004e-05 2020-03-09T00:25:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785681+0000", 2020-03-09T00:25:35.114 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8849999999999999e-06 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785689+0000", 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5290000000000001e-05 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.115 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785704+0000", 2020-03-09T00:25:35.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5759999999999996e-06 2020-03-09T00:25:35.116 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.116 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.116 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.116 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785710+0000", 2020-03-09T00:25:35.116 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012205476 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.797915+0000", 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023467400000000001 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.117 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.798150+0000", 2020-03-09T00:25:35.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00085795900000000004 2020-03-09T00:25:35.118 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.118 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.118 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.118 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799008+0000", 2020-03-09T00:25:35.118 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00050973499999999998 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799518+0000", 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046306960000000001 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.119 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804148+0000", 2020-03-09T00:25:35.120 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00032100500000000002 2020-03-09T00:25:35.120 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.120 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.120 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.120 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804469+0000", 2020-03-09T00:25:35.121 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001 2020-03-09T00:25:35.121 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.121 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.121 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.121 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.121 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.121 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:314 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.294860856, 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998, 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.122 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 314 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.123 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.124 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.124 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:35.124 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9129999999999999e-05 2020-03-09T00:25:35.124 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.124 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.124 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785816+0000", 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1280000000000002e-06 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785823+0000", 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.24e-05 2020-03-09T00:25:35.125 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.126 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785835+0000", 2020-03-09T00:25:35.126 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1800000000000004e-06 2020-03-09T00:25:35.126 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.126 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.126 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785840+0000", 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014080127 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799920+0000", 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023000599999999999 2020-03-09T00:25:35.127 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:35.128 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.800150+0000", 2020-03-09T00:25:35.128 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048975440000000002 2020-03-09T00:25:35.128 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.128 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805048+0000", 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000221747 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805270+0000", 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037657239999999998 2020-03-09T00:25:35.129 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.130 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.130 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.130 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.809036+0000", 2020-03-09T00:25:35.130 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998 2020-03-09T00:25:35.130 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.130 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.130 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.131 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.131 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:35.131 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:35.131 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.229105816999997, 2020-03-09T00:25:35.131 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001, 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.132 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.133 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:35.133 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0030360970000000002 2020-03-09T00:25:35.133 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.133 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.133 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854558+0000", 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8904e-05 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854587+0000", 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5423000000000001e-05 2020-03-09T00:25:35.134 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854602+0000", 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9139999999999997e-06 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.135 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854611+0000", 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2179000000000002e-05 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854673+0000", 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6330000000000001e-05 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.136 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854689+0000", 2020-03-09T00:25:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019248372999999999 2020-03-09T00:25:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873938+0000", 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3464000000000003e-05 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873981+0000", 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.138 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.139 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.139 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.139 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.139 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:397 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~573440] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:35.139 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:35.139 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.252318505000002, 2020-03-09T00:25:35.139 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99949370299999996, 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 397 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.140 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.141 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:35.141 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.141 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.141 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.141 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015862319999999999 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829895+0000", 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1249999999999999e-06 2020-03-09T00:25:35.142 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.143 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829904+0000", 2020-03-09T00:25:35.143 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5704000000000001e-05 2020-03-09T00:25:35.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.143 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.143 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829920+0000", 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3140000000000004e-06 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829928+0000", 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000122717 2020-03-09T00:25:35.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830051+0000", 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9819999999999999e-06 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:35.145 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830056+0000", 2020-03-09T00:25:35.146 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97235390200000005 2020-03-09T00:25:35.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.146 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.146 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.802410+0000", 2020-03-09T00:25:35.146 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016568099999999999 2020-03-09T00:25:35.146 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.147 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.147 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.802576+0000", 2020-03-09T00:25:35.147 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001235402 2020-03-09T00:25:35.147 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.147 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.803811+0000", 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013426433999999999 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817237+0000", 2020-03-09T00:25:35.148 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010489160000000001 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827727+0000", 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6050000000000005e-05 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.149 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.150 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827803+0000", 2020-03-09T00:25:35.150 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99949370299999996 2020-03-09T00:25:35.150 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.150 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.150 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.150 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.150 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:398 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~425984] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.249823061000001, 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99725236100000003, 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.151 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.152 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.152 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 398 2020-03-09T00:25:35.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.152 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.152 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.152 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.152 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:35.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.153 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.153 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.153 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:35.153 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051814600000000004 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831322+0000", 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0118e-05 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.154 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.155 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831333+0000", 2020-03-09T00:25:35.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0710999999999999e-05 2020-03-09T00:25:35.155 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.155 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.155 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831353+0000", 2020-03-09T00:25:35.155 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7870000000000004e-06 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831360+0000", 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013810100000000001 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.156 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831498+0000", 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3380000000000001e-06 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831502+0000", 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97707543200000002 2020-03-09T00:25:35.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.158 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.158 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.808577+0000", 2020-03-09T00:25:35.158 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133669 2020-03-09T00:25:35.158 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.158 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.158 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.808711+0000", 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010978260000000001 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.809808+0000", 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074316110000000003 2020-03-09T00:25:35.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817240+0000", 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010739297 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.160 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827979+0000", 2020-03-09T00:25:35.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7324999999999996e-05 2020-03-09T00:25:35.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.161 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.161 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.161 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828057+0000", 2020-03-09T00:25:35.161 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99725236100000003 2020-03-09T00:25:35.161 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:399 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~344064] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.247721887000001, 2020-03-09T00:25:35.162 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99540426400000004, 2020-03-09T00:25:35.163 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.163 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.163 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.163 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.163 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.163 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 399 2020-03-09T00:25:35.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.164 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00031358200000000002 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833219+0000", 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4809999999999995e-06 2020-03-09T00:25:35.165 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.166 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833229+0000", 2020-03-09T00:25:35.166 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9320000000000008e-06 2020-03-09T00:25:35.166 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.166 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.166 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833238+0000", 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0749999999999999e-06 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833243+0000", 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1554000000000001e-05 2020-03-09T00:25:35.167 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833294+0000", 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4169999999999999e-06 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:35.168 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833297+0000", 2020-03-09T00:25:35.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.979004659 2020-03-09T00:25:35.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.169 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.169 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.169 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812301+0000", 2020-03-09T00:25:35.169 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013289199999999999 2020-03-09T00:25:35.169 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812434+0000", 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098429000000000008 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.170 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813418+0000", 2020-03-09T00:25:35.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0052416019999999997 2020-03-09T00:25:35.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.171 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.171 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.818660+0000", 2020-03-09T00:25:35.171 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095537799999999996 2020-03-09T00:25:35.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828214+0000", 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6000000000000002e-05 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.172 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828310+0000", 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99540426400000004 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:400 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:35.173 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:35.174 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.247348286000001, 2020-03-09T00:25:35.174 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99532314300000002, 2020-03-09T00:25:35.174 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.174 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.174 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.174 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.174 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 400 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9441999999999999e-05 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.176 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833319+0000", 2020-03-09T00:25:35.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.421e-06 2020-03-09T00:25:35.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.177 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.177 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.177 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833325+0000", 2020-03-09T00:25:35.177 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0309999999999998e-06 2020-03-09T00:25:35.177 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833333+0000", 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6189999999999998e-06 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.178 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833336+0000", 2020-03-09T00:25:35.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5015999999999998e-05 2020-03-09T00:25:35.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.179 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.179 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833371+0000", 2020-03-09T00:25:35.179 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.305e-06 2020-03-09T00:25:35.179 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833372+0000", 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.98258547500000004 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.180 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.815957+0000", 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000143335 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816101+0000", 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00076297400000000003 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816864+0000", 2020-03-09T00:25:35.182 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001798628 2020-03-09T00:25:35.182 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.182 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.182 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.182 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.818662+0000", 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0098116109999999996 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828474+0000", 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000128286 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828602+0000", 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99532314300000002 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.184 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.185 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:401 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:35.185 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:35.185 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.247267259000001, 2020-03-09T00:25:35.185 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99818093799999996, 2020-03-09T00:25:35.185 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.185 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:35.185 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.186 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.186 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.186 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 401 2020-03-09T00:25:35.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.186 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.188 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.188 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:35.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.189 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.189 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:35.189 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3272e-05 2020-03-09T00:25:35.189 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833383+0000", 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2030000000000001e-06 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.190 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833387+0000", 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0190000000000006e-06 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833395+0000", 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3800000000000001e-06 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833397+0000", 2020-03-09T00:25:35.192 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9891000000000001e-05 2020-03-09T00:25:35.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.192 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.192 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.192 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833437+0000", 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.564e-06 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833438+0000", 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.98366688700000005 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.193 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817105+0000", 2020-03-09T00:25:35.194 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000121193 2020-03-09T00:25:35.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.194 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.194 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:35.194 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817227+0000", 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011959001 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.829186+0000", 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012604700000000001 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.195 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.829312+0000", 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022294810000000002 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.831541+0000", 2020-03-09T00:25:35.196 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99818093799999996 2020-03-09T00:25:35.197 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.197 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.197 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.197 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.197 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.197 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs2 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:35.197 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 20.277168345, 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019419130999999999, 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.198 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:35.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.199 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.199 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.199 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:35.199 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0024772549999999998 2020-03-09T00:25:35.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805936+0000", 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6125000000000001e-05 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.200 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805962+0000", 2020-03-09T00:25:35.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5769000000000001e-05 2020-03-09T00:25:35.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.201 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.201 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.201 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805978+0000", 2020-03-09T00:25:35.201 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.105e-06 2020-03-09T00:25:35.201 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805984+0000", 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7604e-05 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.202 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.806032+0000", 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2700999999999999e-05 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.806045+0000", 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016799584999999999 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.203 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.822844+0000", 2020-03-09T00:25:35.204 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3986999999999998e-05 2020-03-09T00:25:35.204 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.204 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.204 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.204 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.822878+0000", 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019419130999999999 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:564 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~507904] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:35.205 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:35.206 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.141115231000001, 2020-03-09T00:25:35.206 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016199960999999999, 2020-03-09T00:25:35.206 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.206 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.206 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.206 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.206 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 564 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.207 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.208 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.208 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:35.208 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00049015199999999999 2020-03-09T00:25:35.208 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.208 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.209 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.209 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940002+0000", 2020-03-09T00:25:35.209 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0630000000000006e-06 2020-03-09T00:25:35.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.210 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.210 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940010+0000", 2020-03-09T00:25:35.210 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5739999999999998e-05 2020-03-09T00:25:35.210 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.210 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940026+0000", 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8399999999999997e-06 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940033+0000", 2020-03-09T00:25:35.211 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035392219999999999 2020-03-09T00:25:35.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.212 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.212 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943572+0000", 2020-03-09T00:25:35.212 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8063e-05 2020-03-09T00:25:35.212 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.212 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943660+0000", 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00097790300000000002 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944638+0000", 2020-03-09T00:25:35.213 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039951850000000001 2020-03-09T00:25:35.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.214 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.214 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948633+0000", 2020-03-09T00:25:35.214 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067926059999999996 2020-03-09T00:25:35.214 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.214 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955426+0000", 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00028618699999999998 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955712+0000", 2020-03-09T00:25:35.215 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016199960999999999 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:565 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~622592] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:35.216 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.136972285999999, 2020-03-09T00:25:35.217 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016407618999999998, 2020-03-09T00:25:35.217 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.217 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.217 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.217 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.217 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.217 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 565 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.218 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:35.219 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00060946899999999996 2020-03-09T00:25:35.219 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.219 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.219 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.219 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944265+0000", 2020-03-09T00:25:35.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.004e-05 2020-03-09T00:25:35.220 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.220 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.220 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.220 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944275+0000", 2020-03-09T00:25:35.220 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7229000000000001e-05 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944292+0000", 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4939999999999997e-06 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.221 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944299+0000", 2020-03-09T00:25:35.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046037999999999999 2020-03-09T00:25:35.222 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.222 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.222 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.222 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948903+0000", 2020-03-09T00:25:35.222 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000139479 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.949043+0000", 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011480489999999999 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.223 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950191+0000", 2020-03-09T00:25:35.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055348039999999999 2020-03-09T00:25:35.224 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.224 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.224 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.224 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955725+0000", 2020-03-09T00:25:35.224 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041247109999999997 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.959850+0000", 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000212544 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.225 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960063+0000", 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016407618999999998 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.226 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:566 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~32768] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:35.227 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:35.227 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.136262972000001, 2020-03-09T00:25:35.227 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016519618999999999, 2020-03-09T00:25:35.227 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.227 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.227 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.227 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 566 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:35.228 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.229 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.229 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:35.229 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5429e-05 2020-03-09T00:25:35.229 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.229 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944430+0000", 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2100000000000003e-06 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944434+0000", 2020-03-09T00:25:35.230 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4159999999999998e-06 2020-03-09T00:25:35.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.231 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.231 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.231 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944441+0000", 2020-03-09T00:25:35.231 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3349999999999998e-06 2020-03-09T00:25:35.231 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944445+0000", 2020-03-09T00:25:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0093476589999999995 2020-03-09T00:25:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.232 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953792+0000", 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014473700000000001 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953937+0000", 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00083984299999999995 2020-03-09T00:25:35.233 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.954777+0000", 2020-03-09T00:25:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002244401 2020-03-09T00:25:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.234 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957021+0000", 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035682729999999998 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960590+0000", 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029431599999999998 2020-03-09T00:25:35.235 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960884+0000", 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016519618999999999 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.236 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:567 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.136153266000001, 2020-03-09T00:25:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017056153000000001, 2020-03-09T00:25:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.237 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 567 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.238 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8381999999999999e-05 2020-03-09T00:25:35.239 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944513+0000", 2020-03-09T00:25:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1709999999999998e-06 2020-03-09T00:25:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.240 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944516+0000", 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7849999999999992e-06 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944523+0000", 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2550000000000002e-06 2020-03-09T00:25:35.241 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944527+0000", 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011392539 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.242 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955919+0000", 2020-03-09T00:25:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013716499999999999 2020-03-09T00:25:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956056+0000", 2020-03-09T00:25:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00057513499999999995 2020-03-09T00:25:35.243 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956632+0000", 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00045027700000000001 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.244 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957082+0000", 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042876219999999996 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.961369+0000", 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000160822 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.245 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.961530+0000", 2020-03-09T00:25:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017056153000000001 2020-03-09T00:25:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.246 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.247 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.247 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.247 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:568 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:35.247 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:35.247 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 17.136059795000001, 2020-03-09T00:25:35.247 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020843882000000001, 2020-03-09T00:25:35.247 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.248 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:35.248 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.248 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.248 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.248 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 568 2020-03-09T00:25:35.248 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.248 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.249 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:35.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8635999999999999e-05 2020-03-09T00:25:35.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.250 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.250 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.250 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944626+0000", 2020-03-09T00:25:35.250 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2609999999999999e-06 2020-03-09T00:25:35.250 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.251 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944632+0000", 2020-03-09T00:25:35.251 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4993e-05 2020-03-09T00:25:35.251 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.251 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.251 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944647+0000", 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9439999999999996e-06 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944652+0000", 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013280882000000001 2020-03-09T00:25:35.252 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.253 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957933+0000", 2020-03-09T00:25:35.253 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012477000000000001 2020-03-09T00:25:35.253 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.253 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.253 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:35.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.958058+0000", 2020-03-09T00:25:35.254 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0044866419999999999 2020-03-09T00:25:35.254 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.254 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.254 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.254 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962545+0000", 2020-03-09T00:25:35.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000120438 2020-03-09T00:25:35.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.255 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.255 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.255 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962665+0000", 2020-03-09T00:25:35.255 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027463159999999999 2020-03-09T00:25:35.255 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.965411+0000", 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020843882000000001 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.256 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.257 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.257 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:672 3.6s0 3:6178963a:::smithi10428818-25:head [copy-from ver 45] snapc 2c=[2c,2b,2a,24,23] ondisk+write+known_if_redirected e80)", 2020-03-09T00:25:35.257 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:29.806423+0000", 2020-03-09T00:25:35.257 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.2742048659999998, 2020-03-09T00:25:35.257 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.029706512000000001, 2020-03-09T00:25:35.257 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.257 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 672 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.258 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806423+0000", 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806423+0000", 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2702999999999998e-05 2020-03-09T00:25:35.259 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.260 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806495+0000", 2020-03-09T00:25:35.260 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0570999999999999e-05 2020-03-09T00:25:35.260 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.260 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.260 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806506+0000", 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.588e-05 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806522+0000", 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0649999999999999e-06 2020-03-09T00:25:35.261 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.262 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806530+0000", 2020-03-09T00:25:35.262 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8251000000000003e-05 2020-03-09T00:25:35.262 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.262 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.262 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806578+0000", 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014559299999999999 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806724+0000", 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013694415 2020-03-09T00:25:35.263 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.264 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.820418+0000", 2020-03-09T00:25:35.264 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0089145890000000005 2020-03-09T00:25:35.264 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.264 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.264 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.265 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.829333+0000", 2020-03-09T00:25:35.265 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0066116930000000001 2020-03-09T00:25:35.265 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.265 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.265 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.266 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.835944+0000", 2020-03-09T00:25:35.266 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000184752 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.836129+0000", 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.029706512000000001 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.267 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.268 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.268 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.268 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:673 3.6s0 3:6178963a:::smithi10428818-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e80)", 2020-03-09T00:25:35.268 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:29.806577+0000", 2020-03-09T00:25:35.268 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 5.2740505449999997, 2020-03-09T00:25:35.268 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.030424049000000002, 2020-03-09T00:25:35.268 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.269 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:35.269 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.270 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.270 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.270 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 673 2020-03-09T00:25:35.270 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.272 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.272 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.272 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.272 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806577+0000", 2020-03-09T00:25:35.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.273 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.273 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.273 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.273 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806577+0000", 2020-03-09T00:25:35.273 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4972999999999998e-05 2020-03-09T00:25:35.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.274 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.274 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.274 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806622+0000", 2020-03-09T00:25:35.274 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5189999999999998e-06 2020-03-09T00:25:35.274 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.275 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.275 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.275 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806628+0000", 2020-03-09T00:25:35.275 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2245e-05 2020-03-09T00:25:35.275 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.276 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.276 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806641+0000", 2020-03-09T00:25:35.276 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.78e-06 2020-03-09T00:25:35.276 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.276 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.277 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.277 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806645+0000", 2020-03-09T00:25:35.277 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00025891500000000002 2020-03-09T00:25:35.277 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.277 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.278 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806904+0000", 2020-03-09T00:25:35.278 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4018999999999999e-05 2020-03-09T00:25:35.278 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.278 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.278 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:35.279 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806958+0000", 2020-03-09T00:25:35.279 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022279448 2020-03-09T00:25:35.279 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.279 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.279 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.280 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.829238+0000", 2020-03-09T00:25:35.280 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7843999999999994e-05 2020-03-09T00:25:35.280 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.280 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.280 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:35.280 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.829326+0000", 2020-03-09T00:25:35.281 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074501960000000001 2020-03-09T00:25:35.281 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.281 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.281 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.281 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.836776+0000", 2020-03-09T00:25:35.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3979999999999994e-05 2020-03-09T00:25:35.282 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.282 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.282 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.282 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.836840+0000", 2020-03-09T00:25:35.282 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016113 2020-03-09T00:25:35.283 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.283 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.283 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.283 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.837001+0000", 2020-03-09T00:25:35.283 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.030424049000000002 2020-03-09T00:25:35.283 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.284 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.284 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.284 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.284 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.284 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:762 3.6s0 3:6d582d75:::smithi10428818-10:head [append 0~622592] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:35.284 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:34.890681+0000", 2020-03-09T00:25:35.285 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.189946167, 2020-03-09T00:25:35.285 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016558272999999998, 2020-03-09T00:25:35.285 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.285 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:35.285 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.286 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.286 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.286 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 762 2020-03-09T00:25:35.286 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.286 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.286 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.890681+0000", 2020-03-09T00:25:35.287 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.287 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.287 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.287 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.287 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.890681+0000", 2020-03-09T00:25:35.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00064092800000000003 2020-03-09T00:25:35.288 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.288 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.288 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.288 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891322+0000", 2020-03-09T00:25:35.288 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1939999999999998e-06 2020-03-09T00:25:35.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.289 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.289 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.289 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891330+0000", 2020-03-09T00:25:35.289 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5204999999999999e-05 2020-03-09T00:25:35.289 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.290 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891346+0000", 2020-03-09T00:25:35.290 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9970000000000001e-06 2020-03-09T00:25:35.290 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.290 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.290 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.291 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891353+0000", 2020-03-09T00:25:35.291 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037718790000000001 2020-03-09T00:25:35.291 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.291 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.291 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.291 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.895124+0000", 2020-03-09T00:25:35.291 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013079899999999999 2020-03-09T00:25:35.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.292 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.292 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.895255+0000", 2020-03-09T00:25:35.292 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001217615 2020-03-09T00:25:35.292 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.292 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.293 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.896473+0000", 2020-03-09T00:25:35.293 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0071182620000000002 2020-03-09T00:25:35.293 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.293 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.293 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.903591+0000", 2020-03-09T00:25:35.294 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003383371 2020-03-09T00:25:35.294 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.294 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.294 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:35.294 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.906974+0000", 2020-03-09T00:25:35.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026502299999999997 2020-03-09T00:25:35.295 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.295 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.295 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.295 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.907239+0000", 2020-03-09T00:25:35.295 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016558272999999998 2020-03-09T00:25:35.295 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.296 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.296 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.296 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.296 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.296 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:765 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e85)", 2020-03-09T00:25:35.296 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:34.893621+0000", 2020-03-09T00:25:35.297 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 0.187006318, 2020-03-09T00:25:35.297 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016719357000000001, 2020-03-09T00:25:35.297 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:35.297 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:35.297 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:35.297 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 765 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893621+0000", 2020-03-09T00:25:35.298 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:35.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.299 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:35.299 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893621+0000", 2020-03-09T00:25:35.299 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3843e-05 2020-03-09T00:25:35.299 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.299 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:35.300 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893665+0000", 2020-03-09T00:25:35.300 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8669999999999999e-06 2020-03-09T00:25:35.300 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.300 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.300 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:35.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893671+0000", 2020-03-09T00:25:35.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5889999999999999e-05 2020-03-09T00:25:35.301 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.301 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.301 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:35.301 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893687+0000", 2020-03-09T00:25:35.301 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2579999999999997e-06 2020-03-09T00:25:35.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.302 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.302 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893691+0000", 2020-03-09T00:25:35.302 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010884733000000001 2020-03-09T00:25:35.302 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.302 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.904576+0000", 2020-03-09T00:25:35.303 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000100896 2020-03-09T00:25:35.303 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.303 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.303 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:35.303 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.904677+0000", 2020-03-09T00:25:35.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041684770000000003 2020-03-09T00:25:35.304 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.304 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.304 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:35.304 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.908845+0000", 2020-03-09T00:25:35.304 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012772599999999999 2020-03-09T00:25:35.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.305 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:35.305 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.908973+0000", 2020-03-09T00:25:35.305 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013676669999999999 2020-03-09T00:25:35.305 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:35.305 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:35.306 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:35.306 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.910340+0000", 2020-03-09T00:25:35.306 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016719357000000001 2020-03-09T00:25:35.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.306 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.306 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:35.307 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:35.307 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:35.309 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:25:35.309 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:35.307+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:35.409 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:35.408+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:35.445 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:35.446 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:35.446 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:35.446 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:35.454 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:25:35.510 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:35.508+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:35.578 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:35.578 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:35.579 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:35.579 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:35.587 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:25:35.611 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:35.609+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:35.652 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:35.712 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:35.710+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:35.713 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:35.713 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:35.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:35.713 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:35.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.714 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=15, reqid=client.4751.0:200, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2020-03-09T00:25:35.714 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:25:35.714 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.873326929999998, 2020-03-09T00:25:35.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048492969999999998, 2020-03-09T00:25:35.714 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:25:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837387+0000", 2020-03-09T00:25:35.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026746099999999998 2020-03-09T00:25:35.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837654+0000", 2020-03-09T00:25:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8997e-05 2020-03-09T00:25:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837683+0000", 2020-03-09T00:25:35.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4249999999999999e-05 2020-03-09T00:25:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837717+0000", 2020-03-09T00:25:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9910000000000002e-06 2020-03-09T00:25:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837725+0000", 2020-03-09T00:25:35.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013534699999999999 2020-03-09T00:25:35.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837861+0000", 2020-03-09T00:25:35.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9768e-05 2020-03-09T00:25:35.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837890+0000", 2020-03-09T00:25:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042879290000000002 2020-03-09T00:25:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842178+0000", 2020-03-09T00:25:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7553999999999998e-05 2020-03-09T00:25:35.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842236+0000", 2020-03-09T00:25:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048492969999999998 2020-03-09T00:25:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.722 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.723 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:35.723 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:35.723 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.860313570999999, 2020-03-09T00:25:35.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057419319999999999, 2020-03-09T00:25:35.723 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:35.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056030900000000005 2020-03-09T00:25:35.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850960+0000", 2020-03-09T00:25:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9046e-05 2020-03-09T00:25:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850979+0000", 2020-03-09T00:25:35.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0473e-05 2020-03-09T00:25:35.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850990+0000", 2020-03-09T00:25:35.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.198e-06 2020-03-09T00:25:35.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850995+0000", 2020-03-09T00:25:35.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123569 2020-03-09T00:25:35.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.851119+0000", 2020-03-09T00:25:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4375e-05 2020-03-09T00:25:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.851133+0000", 2020-03-09T00:25:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049526300000000004 2020-03-09T00:25:35.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856086+0000", 2020-03-09T00:25:35.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6332000000000002e-05 2020-03-09T00:25:35.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856142+0000", 2020-03-09T00:25:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057419319999999999 2020-03-09T00:25:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.846725610999997, 2020-03-09T00:25:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060463440000000004, 2020-03-09T00:25:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088386200000000002 2020-03-09T00:25:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864872+0000", 2020-03-09T00:25:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2838e-05 2020-03-09T00:25:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864905+0000", 2020-03-09T00:25:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9593e-05 2020-03-09T00:25:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864924+0000", 2020-03-09T00:25:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8414999999999999e-05 2020-03-09T00:25:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864943+0000", 2020-03-09T00:25:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000227028 2020-03-09T00:25:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865170+0000", 2020-03-09T00:25:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7546000000000001e-05 2020-03-09T00:25:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865187+0000", 2020-03-09T00:25:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047657489999999997 2020-03-09T00:25:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.869953+0000", 2020-03-09T00:25:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1312999999999999e-05 2020-03-09T00:25:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.870034+0000", 2020-03-09T00:25:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060463440000000004 2020-03-09T00:25:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/32 ECSubWrite(tid=22, reqid=client.4751.0:275, at_version=48'12, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:25:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.922787890999999, 2020-03-09T00:25:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080719139999999995, 2020-03-09T00:25:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013854899999999999 2020-03-09T00:25:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788064+0000", 2020-03-09T00:25:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9998000000000001e-05 2020-03-09T00:25:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788084+0000", 2020-03-09T00:25:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2672e-05 2020-03-09T00:25:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788097+0000", 2020-03-09T00:25:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5619999999999998e-06 2020-03-09T00:25:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788104+0000", 2020-03-09T00:25:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0817999999999999e-05 2020-03-09T00:25:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788165+0000", 2020-03-09T00:25:35.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9372999999999999e-05 2020-03-09T00:25:35.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788185+0000", 2020-03-09T00:25:35.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077469949999999996 2020-03-09T00:25:35.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.749 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795932+0000", 2020-03-09T00:25:35.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5946999999999997e-05 2020-03-09T00:25:35.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.749 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795998+0000", 2020-03-09T00:25:35.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080719139999999995 2020-03-09T00:25:35.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.750 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.751 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:35.751 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:35.751 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.893626808999997, 2020-03-09T00:25:35.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052768320000000004, 2020-03-09T00:25:35.751 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.752 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.752 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:35.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:35.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040852899999999998 2020-03-09T00:25:35.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817495+0000", 2020-03-09T00:25:35.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1301000000000002e-05 2020-03-09T00:25:35.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817517+0000", 2020-03-09T00:25:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4061e-05 2020-03-09T00:25:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817531+0000", 2020-03-09T00:25:35.755 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0720000000000001e-06 2020-03-09T00:25:35.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.756 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.756 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817535+0000", 2020-03-09T00:25:35.756 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012169500000000001 2020-03-09T00:25:35.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.757 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817656+0000", 2020-03-09T00:25:35.757 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3953e-05 2020-03-09T00:25:35.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.757 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.757 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817670+0000", 2020-03-09T00:25:35.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046400950000000003 2020-03-09T00:25:35.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.758 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.758 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822310+0000", 2020-03-09T00:25:35.758 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3125999999999999e-05 2020-03-09T00:25:35.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.759 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.759 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822364+0000", 2020-03-09T00:25:35.759 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052768320000000004 2020-03-09T00:25:35.759 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.760 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.760 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.760 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.760 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.760 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 54/32 ECSubWrite(tid=77, reqid=client.4751.0:415, at_version=54'44, trim_to=0'0, roll_forward_to=54'42))", 2020-03-09T00:25:35.760 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:35.760 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.507771326, 2020-03-09T00:25:35.761 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066307969999999999, 2020-03-09T00:25:35.761 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.761 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.761 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.761 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.761 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:35.762 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.762 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.762 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:35.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055621700000000002 2020-03-09T00:25:35.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.763 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.763 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.763 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203498+0000", 2020-03-09T00:25:35.763 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0248999999999999e-05 2020-03-09T00:25:35.763 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.764 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.764 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203519+0000", 2020-03-09T00:25:35.764 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3159000000000001e-05 2020-03-09T00:25:35.764 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.764 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203532+0000", 2020-03-09T00:25:35.765 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8510000000000004e-06 2020-03-09T00:25:35.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.765 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.765 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203538+0000", 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1504e-05 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203589+0000", 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5809000000000001e-05 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.767 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203605+0000", 2020-03-09T00:25:35.767 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059129430000000004 2020-03-09T00:25:35.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.767 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.767 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.209518+0000", 2020-03-09T00:25:35.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5065000000000002e-05 2020-03-09T00:25:35.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.768 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.768 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.209573+0000", 2020-03-09T00:25:35.768 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066307969999999999 2020-03-09T00:25:35.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.769 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.769 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s3 59/54 ECSubWrite(tid=164, reqid=client.4751.0:490, at_version=59'45, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:35.770 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.836818+0000", 2020-03-09T00:25:35.770 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.873895544, 2020-03-09T00:25:35.770 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049356169999999998, 2020-03-09T00:25:35.770 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.770 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.770 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.771 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.836818+0000", 2020-03-09T00:25:35.771 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.771 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.771 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.836818+0000", 2020-03-09T00:25:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059775900000000001 2020-03-09T00:25:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837416+0000", 2020-03-09T00:25:35.772 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7926000000000001e-05 2020-03-09T00:25:35.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.773 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.773 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837434+0000", 2020-03-09T00:25:35.773 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8789999999999992e-06 2020-03-09T00:25:35.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.774 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.774 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837443+0000", 2020-03-09T00:25:35.774 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4440000000000001e-06 2020-03-09T00:25:35.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.774 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837447+0000", 2020-03-09T00:25:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2846999999999998e-05 2020-03-09T00:25:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.775 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837490+0000", 2020-03-09T00:25:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1929e-05 2020-03-09T00:25:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837502+0000", 2020-03-09T00:25:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042011940000000001 2020-03-09T00:25:35.776 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.841703+0000", 2020-03-09T00:25:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0639000000000001e-05 2020-03-09T00:25:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.777 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.841754+0000", 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049356169999999998 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.778 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s3 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.843432+0000", 2020-03-09T00:25:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.867281465000001, 2020-03-09T00:25:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052060020000000004, 2020-03-09T00:25:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.779 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.780 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.780 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843432+0000", 2020-03-09T00:25:35.780 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843432+0000", 2020-03-09T00:25:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011910670000000001 2020-03-09T00:25:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.781 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844623+0000", 2020-03-09T00:25:35.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5556999999999999e-05 2020-03-09T00:25:35.782 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.782 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.782 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.782 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844649+0000", 2020-03-09T00:25:35.782 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9201000000000001e-05 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844668+0000", 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2920000000000001e-06 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.783 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844674+0000", 2020-03-09T00:25:35.784 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4917000000000002e-05 2020-03-09T00:25:35.784 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.784 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.784 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.784 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844739+0000", 2020-03-09T00:25:35.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6398000000000001e-05 2020-03-09T00:25:35.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.785 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.785 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.785 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.844756+0000", 2020-03-09T00:25:35.785 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038280250000000001 2020-03-09T00:25:35.785 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.848584+0000", 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4545e-05 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.786 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.848638+0000", 2020-03-09T00:25:35.787 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052060020000000004 2020-03-09T00:25:35.787 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.787 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.787 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.787 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.787 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.796605496000002, 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094365569999999999, 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:35.789 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.789 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.789 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.789 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.789 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011685599999999999 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915277+0000", 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3901999999999999e-05 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.790 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915301+0000", 2020-03-09T00:25:35.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4554e-05 2020-03-09T00:25:35.791 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.791 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.791 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915315+0000", 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8169999999999999e-06 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915323+0000", 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2163999999999998e-05 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915365+0000", 2020-03-09T00:25:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4710000000000001e-06 2020-03-09T00:25:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915373+0000", 2020-03-09T00:25:35.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081172520000000001 2020-03-09T00:25:35.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923490+0000", 2020-03-09T00:25:35.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4836999999999997e-05 2020-03-09T00:25:35.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923545+0000", 2020-03-09T00:25:35.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094365569999999999 2020-03-09T00:25:35.795 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.795 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.795 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.796 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs2 68/67 ECSubWrite(tid=91, reqid=client.4751.0:580, at_version=68'59, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:35.796 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.967188+0000", 2020-03-09T00:25:35.796 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.743525562999999, 2020-03-09T00:25:35.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058375959999999996, 2020-03-09T00:25:35.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967188+0000", 2020-03-09T00:25:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967188+0000", 2020-03-09T00:25:35.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00078212699999999997 2020-03-09T00:25:35.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967970+0000", 2020-03-09T00:25:35.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4190999999999999e-05 2020-03-09T00:25:35.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967994+0000", 2020-03-09T00:25:35.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6008e-05 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968010+0000", 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6010000000000002e-06 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968016+0000", 2020-03-09T00:25:35.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3309999999999997e-05 2020-03-09T00:25:35.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968059+0000", 2020-03-09T00:25:35.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3348e-05 2020-03-09T00:25:35.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968073+0000", 2020-03-09T00:25:35.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048942650000000001 2020-03-09T00:25:35.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.972967+0000", 2020-03-09T00:25:35.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8746000000000003e-05 2020-03-09T00:25:35.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973026+0000", 2020-03-09T00:25:35.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058375959999999996 2020-03-09T00:25:35.804 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.804 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.804 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.804 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s1 68/67 ECSubWrite(tid=237, reqid=client.4751.0:583, at_version=68'129, trim_to=0'0, roll_forward_to=68'126))", 2020-03-09T00:25:35.805 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.973084+0000", 2020-03-09T00:25:35.805 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.737629987999998, 2020-03-09T00:25:35.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077488779999999998, 2020-03-09T00:25:35.805 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.805 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.805 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973084+0000", 2020-03-09T00:25:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973084+0000", 2020-03-09T00:25:35.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020823799999999999 2020-03-09T00:25:35.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973292+0000", 2020-03-09T00:25:35.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9747000000000002e-05 2020-03-09T00:25:35.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.810 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973322+0000", 2020-03-09T00:25:35.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1814000000000002e-05 2020-03-09T00:25:35.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.810 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973343+0000", 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8770000000000005e-06 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973353+0000", 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7701000000000002e-05 2020-03-09T00:25:35.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973411+0000", 2020-03-09T00:25:35.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5305999999999999e-05 2020-03-09T00:25:35.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973426+0000", 2020-03-09T00:25:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073632940000000003 2020-03-09T00:25:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980790+0000", 2020-03-09T00:25:35.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2901000000000001e-05 2020-03-09T00:25:35.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980832+0000", 2020-03-09T00:25:35.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077488779999999998 2020-03-09T00:25:35.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.815 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.815 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.2s5 68/48 ECSubWrite(tid=36, reqid=client.4751.0:587, at_version=68'43, trim_to=0'0, roll_forward_to=67'41))", 2020-03-09T00:25:35.815 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.979226+0000", 2020-03-09T00:25:35.815 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.731487157, 2020-03-09T00:25:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067534049999999997, 2020-03-09T00:25:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979226+0000", 2020-03-09T00:25:35.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979226+0000", 2020-03-09T00:25:35.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013291099999999999 2020-03-09T00:25:35.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979359+0000", 2020-03-09T00:25:35.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4303000000000001e-05 2020-03-09T00:25:35.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979384+0000", 2020-03-09T00:25:35.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8671000000000001e-05 2020-03-09T00:25:35.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979402+0000", 2020-03-09T00:25:35.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6800000000000005e-06 2020-03-09T00:25:35.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979412+0000", 2020-03-09T00:25:35.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6808999999999998e-05 2020-03-09T00:25:35.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979469+0000", 2020-03-09T00:25:35.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2595000000000001e-05 2020-03-09T00:25:35.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979481+0000", 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064418380000000001 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985923+0000", 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6598000000000003e-05 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985980+0000", 2020-03-09T00:25:35.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067534049999999997 2020-03-09T00:25:35.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.823 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.2s5 68/48 ECSubWrite(tid=38, reqid=client.4751.0:589, at_version=68'45, trim_to=0'0, roll_forward_to=67'41))", 2020-03-09T00:25:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.987587+0000", 2020-03-09T00:25:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.723126511, 2020-03-09T00:25:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047998470000000003, 2020-03-09T00:25:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.824 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987587+0000", 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987587+0000", 2020-03-09T00:25:35.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108758 2020-03-09T00:25:35.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987696+0000", 2020-03-09T00:25:35.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8467000000000003e-05 2020-03-09T00:25:35.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987734+0000", 2020-03-09T00:25:35.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7356000000000001e-05 2020-03-09T00:25:35.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987752+0000", 2020-03-09T00:25:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0689000000000001e-05 2020-03-09T00:25:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987772+0000", 2020-03-09T00:25:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059190999999999996 2020-03-09T00:25:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.988364+0000", 2020-03-09T00:25:35.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4902000000000001e-05 2020-03-09T00:25:35.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.988379+0000", 2020-03-09T00:25:35.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039482120000000004 2020-03-09T00:25:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992327+0000", 2020-03-09T00:25:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9552999999999997e-05 2020-03-09T00:25:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992387+0000", 2020-03-09T00:25:35.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047998470000000003 2020-03-09T00:25:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.834 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 68/67 ECSubWrite(tid=101, reqid=client.4751.0:592, at_version=68'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:35.834 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.989023+0000", 2020-03-09T00:25:35.834 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.721690957, 2020-03-09T00:25:35.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079003479999999997, 2020-03-09T00:25:35.834 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.834 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.834 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989023+0000", 2020-03-09T00:25:35.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989023+0000", 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064198599999999999 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989665+0000", 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6575e-05 2020-03-09T00:25:35.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989681+0000", 2020-03-09T00:25:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0254e-05 2020-03-09T00:25:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989701+0000", 2020-03-09T00:25:35.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8129999999999998e-06 2020-03-09T00:25:35.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989708+0000", 2020-03-09T00:25:35.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7768000000000001e-05 2020-03-09T00:25:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989756+0000", 2020-03-09T00:25:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5208999999999999e-05 2020-03-09T00:25:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989781+0000", 2020-03-09T00:25:35.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070914699999999999 2020-03-09T00:25:35.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996873+0000", 2020-03-09T00:25:35.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0272999999999998e-05 2020-03-09T00:25:35.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996923+0000", 2020-03-09T00:25:35.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079003479999999997 2020-03-09T00:25:35.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.842 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.842 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 68/67 ECSubWrite(tid=102, reqid=client.4751.0:593, at_version=68'24, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:35.843 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.993168+0000", 2020-03-09T00:25:35.843 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.717545728000001, 2020-03-09T00:25:35.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068995819999999996, 2020-03-09T00:25:35.843 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.843 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.843 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993168+0000", 2020-03-09T00:25:35.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993168+0000", 2020-03-09T00:25:35.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154976 2020-03-09T00:25:35.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993323+0000", 2020-03-09T00:25:35.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6016e-05 2020-03-09T00:25:35.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993349+0000", 2020-03-09T00:25:35.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8249000000000001e-05 2020-03-09T00:25:35.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993367+0000", 2020-03-09T00:25:35.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2140000000000001e-06 2020-03-09T00:25:35.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993373+0000", 2020-03-09T00:25:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016447 2020-03-09T00:25:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993538+0000", 2020-03-09T00:25:35.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5218000000000001e-05 2020-03-09T00:25:35.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993553+0000", 2020-03-09T00:25:35.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064617299999999997 2020-03-09T00:25:35.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.851 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:18.000015+0000", 2020-03-09T00:25:35.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2709e-05 2020-03-09T00:25:35.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:18.000067+0000", 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068995819999999996 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.852 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s3 75/67 ECSubWrite(tid=93, reqid=client.4751.0:637, at_version=75'8, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:35.853 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:24.810094+0000", 2020-03-09T00:25:35.853 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.900620074000001, 2020-03-09T00:25:35.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015640494000000001, 2020-03-09T00:25:35.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.853 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.854 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.810094+0000", 2020-03-09T00:25:35.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.810094+0000", 2020-03-09T00:25:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022108739999999998 2020-03-09T00:25:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812304+0000", 2020-03-09T00:25:35.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4171999999999997e-05 2020-03-09T00:25:35.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812339+0000", 2020-03-09T00:25:35.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2938e-05 2020-03-09T00:25:35.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812351+0000", 2020-03-09T00:25:35.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.221e-06 2020-03-09T00:25:35.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812359+0000", 2020-03-09T00:25:35.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5735000000000007e-05 2020-03-09T00:25:35.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812424+0000", 2020-03-09T00:25:35.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5446000000000001e-05 2020-03-09T00:25:35.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812440+0000", 2020-03-09T00:25:35.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013249169999999999 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.825689+0000", 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4938000000000002e-05 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.825734+0000", 2020-03-09T00:25:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015640494000000001 2020-03-09T00:25:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:29.821237+0000", 2020-03-09T00:25:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.8894764930000001, 2020-03-09T00:25:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089333529999999998, 2020-03-09T00:25:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821237+0000", 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821237+0000", 2020-03-09T00:25:35.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010328519999999999 2020-03-09T00:25:35.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822270+0000", 2020-03-09T00:25:35.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2592999999999999e-05 2020-03-09T00:25:35.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822293+0000", 2020-03-09T00:25:35.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2020-03-09T00:25:35.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822304+0000", 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4410000000000001e-06 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822309+0000", 2020-03-09T00:25:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2065999999999999e-05 2020-03-09T00:25:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822361+0000", 2020-03-09T00:25:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7379000000000001e-05 2020-03-09T00:25:35.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822378+0000", 2020-03-09T00:25:35.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077357290000000002 2020-03-09T00:25:35.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.830114+0000", 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6314999999999999e-05 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.830170+0000", 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089333529999999998 2020-03-09T00:25:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.871 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.871 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s3 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:35.871 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.884858+0000", 2020-03-09T00:25:35.871 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.825856064, 2020-03-09T00:25:35.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018966332999999998, 2020-03-09T00:25:35.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.872 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.873 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.884858+0000", 2020-03-09T00:25:35.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.884858+0000", 2020-03-09T00:25:35.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039268819999999996 2020-03-09T00:25:35.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888784+0000", 2020-03-09T00:25:35.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9009999999999999e-05 2020-03-09T00:25:35.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888803+0000", 2020-03-09T00:25:35.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2874000000000001e-05 2020-03-09T00:25:35.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888816+0000", 2020-03-09T00:25:35.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8810000000000004e-06 2020-03-09T00:25:35.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888821+0000", 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4030999999999999e-05 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888865+0000", 2020-03-09T00:25:35.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2290000000000003e-06 2020-03-09T00:25:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888873+0000", 2020-03-09T00:25:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014886276 2020-03-09T00:25:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.903760+0000", 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4150000000000001e-05 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.903824+0000", 2020-03-09T00:25:35.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018966332999999998 2020-03-09T00:25:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.881 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 85/67 ECSubWrite(tid=124, reqid=client.4751.0:762, at_version=85'38, trim_to=0'0, roll_forward_to=85'37))", 2020-03-09T00:25:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.896639+0000", 2020-03-09T00:25:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.81407478899999997, 2020-03-09T00:25:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099964359999999992, 2020-03-09T00:25:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896639+0000", 2020-03-09T00:25:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896639+0000", 2020-03-09T00:25:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052756099999999996 2020-03-09T00:25:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897166+0000", 2020-03-09T00:25:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7646999999999999e-05 2020-03-09T00:25:35.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897184+0000", 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0543000000000001e-05 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897195+0000", 2020-03-09T00:25:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0690000000000004e-06 2020-03-09T00:25:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897202+0000", 2020-03-09T00:25:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034302289999999999 2020-03-09T00:25:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900632+0000", 2020-03-09T00:25:35.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.378e-05 2020-03-09T00:25:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900646+0000", 2020-03-09T00:25:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059260010000000002 2020-03-09T00:25:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906572+0000", 2020-03-09T00:25:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3606000000000001e-05 2020-03-09T00:25:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906635+0000", 2020-03-09T00:25:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099964359999999992 2020-03-09T00:25:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.890 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 85/67 ECSubWrite(tid=126, reqid=client.4751.0:763, at_version=85'39, trim_to=0'0, roll_forward_to=85'37))", 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.901520+0000", 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.80919334399999998, 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052168550000000003, 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:35.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901520+0000", 2020-03-09T00:25:35.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:35.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:35.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901520+0000", 2020-03-09T00:25:35.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000133857 2020-03-09T00:25:35.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:35.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901654+0000", 2020-03-09T00:25:35.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5188999999999999e-05 2020-03-09T00:25:35.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901679+0000", 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7649e-05 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901697+0000", 2020-03-09T00:25:35.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0185e-05 2020-03-09T00:25:35.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:35.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901707+0000", 2020-03-09T00:25:35.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028222220000000001 2020-03-09T00:25:35.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:35.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.904529+0000", 2020-03-09T00:25:35.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2162e-05 2020-03-09T00:25:35.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.904542+0000", 2020-03-09T00:25:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021478690000000002 2020-03-09T00:25:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906689+0000", 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7722000000000002e-05 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906737+0000", 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052168550000000003 2020-03-09T00:25:35.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.899 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.899 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.899 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:35.899 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:35.899 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:35.901 INFO:tasks.rados.rados.0.smithi104.stdout:300: done (3 left) 2020-03-09T00:25:35.902 INFO:tasks.rados.rados.0.smithi104.stdout:301: done (2 left) 2020-03-09T00:25:35.902 INFO:tasks.rados.rados.0.smithi104.stdout:302: done (1 left) 2020-03-09T00:25:35.902 INFO:tasks.rados.rados.0.smithi104.stdout:307: done (0 left) 2020-03-09T00:25:35.902 INFO:tasks.rados.rados.0.smithi104.stdout:308: copy_from oid 31 from oid 19 current snap is 25 2020-03-09T00:25:35.902 INFO:tasks.rados.rados.0.smithi104.stdout:309: snap_create 2020-03-09T00:25:35.902 INFO:tasks.rados.rados.0.smithi104.stdout:308: finishing copy_from to smithi10428818-31 2020-03-09T00:25:35.903 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 31 v 24 (ObjNum 75 snap 11 seq_num 75) dirty exists 2020-03-09T00:25:35.903 INFO:tasks.rados.rados.0.smithi104.stdout:308: finishing copy_from racing read to smithi10428818-31 2020-03-09T00:25:35.903 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:35.903 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:35.903 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:35.903 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:35.904 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:35.904 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:35.899+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:35.904 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:25:35.915 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:35.938 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:35.958 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:35.979 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:36.000 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:36.001 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:35.999+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:36.021 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:36.025 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.025 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.040 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:25:36.042 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:36.063 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:36.068 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-03-09T00:25:36.087 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:36.102 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:36.101+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:36.110 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:36.111 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.111 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.111 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.111 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.111 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:36.121 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:36.162 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.162 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.172 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:25:36.205 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:36.203+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:36.297 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.298 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.307 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:36.305+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:36.307 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:25:36.407 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:36.406+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:36.433 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.433 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.442 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:25:36.508 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:36.506+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:36.556 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.557 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.565 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:25:36.609 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:36.608+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:36.710 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:36.708+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:36.720 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.720 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.729 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:25:36.812 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:36.810+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:36.829 INFO:tasks.rados.rados.0.smithi104.stdout:308: done (1 left) 2020-03-09T00:25:36.829 INFO:tasks.rados.rados.0.smithi104.stdout:309: done (0 left) 2020-03-09T00:25:36.829 INFO:tasks.rados.rados.0.smithi104.stdout:310: delete oid 3 current snap is 26 2020-03-09T00:25:36.832 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is unset 2020-03-09T00:25:36.836 INFO:tasks.rados.rados.0.smithi104.stdout:310: done (0 left) 2020-03-09T00:25:36.836 INFO:tasks.rados.rados.0.smithi104.stdout:311: read oid 20 snap 25 2020-03-09T00:25:36.837 INFO:tasks.rados.rados.0.smithi104.stdout:311: expect deleted 2020-03-09T00:25:36.837 INFO:tasks.rados.rados.0.smithi104.stdout:312: rollback oid 44 current snap is 26 2020-03-09T00:25:36.837 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 44 to 25 2020-03-09T00:25:36.837 INFO:tasks.rados.rados.0.smithi104.stdout:313: rollback oid 7 current snap is 26 2020-03-09T00:25:36.837 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 7 to 25 2020-03-09T00:25:36.837 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 41 current snap is 26 2020-03-09T00:25:36.838 INFO:tasks.rados.rados.0.smithi104.stdout:314: seq_num 130 ranges {688128=638976,1327104=671744,1998848=802816,2801664=81920} 2020-03-09T00:25:36.844 INFO:tasks.rados.rados.0.smithi104.stdout:314: writing smithi10428818-41 from 688128 to 1327104 tid 1 2020-03-09T00:25:36.848 INFO:tasks.rados.rados.0.smithi104.stdout:314: writing smithi10428818-41 from 1327104 to 1998848 tid 2 2020-03-09T00:25:36.853 INFO:tasks.rados.rados.0.smithi104.stdout:314: writing smithi10428818-41 from 1998848 to 2801664 tid 3 2020-03-09T00:25:36.854 INFO:tasks.rados.rados.0.smithi104.stdout:314: writing smithi10428818-41 from 2801664 to 2883584 tid 4 2020-03-09T00:25:36.854 INFO:tasks.rados.rados.0.smithi104.stdout:315: read oid 45 snap 25 2020-03-09T00:25:36.854 INFO:tasks.rados.rados.0.smithi104.stdout:315: expect (ObjNum 44 snap 0 seq_num 44) 2020-03-09T00:25:36.854 INFO:tasks.rados.rados.0.smithi104.stdout:316: snap_remove snap 24 2020-03-09T00:25:36.863 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.864 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.873 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:25:36.913 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:36.912+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:36.988 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:36.988 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:36.996 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:25:37.041 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:37.040+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:37.113 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:37.113 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:37.115 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:37.114+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:37.121 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:25:37.217 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:37.215+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:37.258 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:37.259 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:37.268 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:25:37.317 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:37.316+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:37.392 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:37.392 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:37.401 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:25:37.442 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:37.441+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:37.519 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:37.518+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:37.566 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:37.566 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:37.575 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:25:37.621 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:37.619+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:37.697 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:37.697 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:37.706 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:25:37.746 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:37.745+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:37.822 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:37.821+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:37.836 INFO:tasks.rados.rados.0.smithi104.stdout:316: done (5 left) 2020-03-09T00:25:37.836 INFO:tasks.rados.rados.0.smithi104.stdout:317: read oid 48 snap -1 2020-03-09T00:25:37.837 INFO:tasks.rados.rados.0.smithi104.stdout:317: expect (ObjNum 71 snap 11 seq_num 71) 2020-03-09T00:25:37.837 INFO:tasks.rados.rados.0.smithi104.stdout:318: copy_from oid 39 from oid 29 current snap is 26 2020-03-09T00:25:37.837 INFO:tasks.rados.rados.0.smithi104.stdout:312: finishing rollback tid 0 to smithi10428818-44 2020-03-09T00:25:37.837 INFO:tasks.rados.rados.0.smithi104.stdout:311: done (6 left) 2020-03-09T00:25:37.838 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 15 current snap is 26 2020-03-09T00:25:37.838 INFO:tasks.rados.rados.0.smithi104.stdout:319: seq_num 131 ranges {1884160=655360,2539520=425984,2965504=638976,3604480=491520} 2020-03-09T00:25:37.851 INFO:tasks.rados.rados.0.smithi104.stdout:319: writing smithi10428818-15 from 1884160 to 2539520 tid 1 2020-03-09T00:25:37.853 INFO:tasks.rados.rados.0.smithi104.stdout:319: writing smithi10428818-15 from 2539520 to 2965504 tid 2 2020-03-09T00:25:37.857 INFO:tasks.rados.rados.0.smithi104.stdout:319: writing smithi10428818-15 from 2965504 to 3604480 tid 3 2020-03-09T00:25:37.860 INFO:tasks.rados.rados.0.smithi104.stdout:319: writing smithi10428818-15 from 3604480 to 4096000 tid 4 2020-03-09T00:25:37.860 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 47 current snap is 26 2020-03-09T00:25:37.860 INFO:tasks.rados.rados.0.smithi104.stdout:320: seq_num 132 ranges {0=671744,671744=573440,1245184=753664,1998848=229376} 2020-03-09T00:25:37.865 INFO:tasks.rados.rados.0.smithi104.stdout:320: writing smithi10428818-47 from 0 to 671744 tid 1 2020-03-09T00:25:37.867 INFO:tasks.rados.rados.0.smithi104.stdout:320: writing smithi10428818-47 from 671744 to 1245184 tid 2 2020-03-09T00:25:37.871 INFO:tasks.rados.rados.0.smithi104.stdout:320: writing smithi10428818-47 from 1245184 to 1998848 tid 3 2020-03-09T00:25:37.872 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:37.872 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:37.874 INFO:tasks.rados.rados.0.smithi104.stdout:320: writing smithi10428818-47 from 1998848 to 2228224 tid 4 2020-03-09T00:25:37.874 INFO:tasks.rados.rados.0.smithi104.stdout:321: rollback oid 30 current snap is 26 2020-03-09T00:25:37.875 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 30 to 25 2020-03-09T00:25:37.875 INFO:tasks.rados.rados.0.smithi104.stdout:313: finishing rollback tid 0 to smithi10428818-7 2020-03-09T00:25:37.875 INFO:tasks.rados.rados.0.smithi104.stdout:313: finishing rollback tid 1 to smithi10428818-7 2020-03-09T00:25:37.876 INFO:tasks.rados.rados.0.smithi104.stdout:313: finishing rollback tid 2 to smithi10428818-7 2020-03-09T00:25:37.876 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 7 v 162 (ObjNum 117 snap 24 seq_num 117) dirty exists 2020-03-09T00:25:37.876 INFO:tasks.rados.rados.0.smithi104.stdout:312: finishing rollback tid 1 to smithi10428818-44 2020-03-09T00:25:37.876 INFO:tasks.rados.rados.0.smithi104.stdout:312: finishing rollback tid 2 to smithi10428818-44 2020-03-09T00:25:37.877 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 44 v 89 (ObjNum 116 snap 24 seq_num 116) dirty exists 2020-03-09T00:25:37.877 INFO:tasks.rados.rados.0.smithi104.stdout:315: expect (ObjNum 44 snap 0 seq_num 44) 2020-03-09T00:25:37.886 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:25:37.900 INFO:tasks.rados.rados.0.smithi104.stdout:314: finishing write tid 1 to smithi10428818-41 2020-03-09T00:25:37.901 INFO:tasks.rados.rados.0.smithi104.stdout:314: finishing write tid 2 to smithi10428818-41 2020-03-09T00:25:37.901 INFO:tasks.rados.rados.0.smithi104.stdout:314: finishing write tid 3 to smithi10428818-41 2020-03-09T00:25:37.901 INFO:tasks.rados.rados.0.smithi104.stdout:314: finishing write tid 4 to smithi10428818-41 2020-03-09T00:25:37.901 INFO:tasks.rados.rados.0.smithi104.stdout:314: finishing write tid 6 to smithi10428818-41 2020-03-09T00:25:37.902 INFO:tasks.rados.rados.0.smithi104.stdout:314: finishing write tid 7 to smithi10428818-41 2020-03-09T00:25:37.902 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 41 v 35 (ObjNum 130 snap 26 seq_num 130) dirty exists 2020-03-09T00:25:37.902 INFO:tasks.rados.rados.0.smithi104.stdout:314: left oid 41 (ObjNum 130 snap 26 seq_num 130) 2020-03-09T00:25:37.902 INFO:tasks.rados.rados.0.smithi104.stdout:312: done (8 left) 2020-03-09T00:25:37.903 INFO:tasks.rados.rados.0.smithi104.stdout:313: done (7 left) 2020-03-09T00:25:37.905 INFO:tasks.rados.rados.0.smithi104.stdout:314: done (6 left) 2020-03-09T00:25:37.905 INFO:tasks.rados.rados.0.smithi104.stdout:315: done (5 left) 2020-03-09T00:25:37.906 INFO:tasks.rados.rados.0.smithi104.stdout:322: rollback oid 40 current snap is 26 2020-03-09T00:25:37.906 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 40 to 25 2020-03-09T00:25:37.906 INFO:tasks.rados.rados.0.smithi104.stdout:320: finishing write tid 1 to smithi10428818-47 2020-03-09T00:25:37.906 INFO:tasks.rados.rados.0.smithi104.stdout:317: expect (ObjNum 71 snap 11 seq_num 71) 2020-03-09T00:25:37.930 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:37.929+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:38.008 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:38.009 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:38.016 INFO:tasks.rados.rados.0.smithi104.stdout:321: finishing rollback tid 1 to smithi10428818-30 2020-03-09T00:25:38.016 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 30 v 50 (ObjNum 1 snap 32510 seq_num 3967828832) dirty dne 2020-03-09T00:25:38.016 INFO:tasks.rados.rados.0.smithi104.stdout:318: finishing copy_from to smithi10428818-39 2020-03-09T00:25:38.016 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 39 v 29 (ObjNum 80 snap 12 seq_num 80) dirty exists 2020-03-09T00:25:38.016 INFO:tasks.rados.rados.0.smithi104.stdout:318: finishing copy_from racing read to smithi10428818-39 2020-03-09T00:25:38.016 INFO:tasks.rados.rados.0.smithi104.stdout:319: finishing write tid 1 to smithi10428818-15 2020-03-09T00:25:38.016 INFO:tasks.rados.rados.0.smithi104.stdout:319: finishing write tid 2 to smithi10428818-15 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:320: finishing write tid 2 to smithi10428818-47 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:317: done (5 left) 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:318: done (4 left) 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:321: done (3 left) 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:323: delete oid 34 current snap is 26 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:319: finishing write tid 3 to smithi10428818-15 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:320: finishing write tid 3 to smithi10428818-47 2020-03-09T00:25:38.017 INFO:tasks.rados.rados.0.smithi104.stdout:319: finishing write tid 4 to smithi10428818-15 2020-03-09T00:25:38.018 INFO:tasks.rados.rados.0.smithi104.stdout:320: finishing write tid 4 to smithi10428818-47 2020-03-09T00:25:38.018 INFO:tasks.rados.rados.0.smithi104.stdout:319: finishing write tid 6 to smithi10428818-15 2020-03-09T00:25:38.018 INFO:tasks.rados.rados.0.smithi104.stdout:320: finishing write tid 6 to smithi10428818-47 2020-03-09T00:25:38.018 INFO:tasks.rados.rados.0.smithi104.stdout:319: finishing write tid 7 to smithi10428818-15 2020-03-09T00:25:38.018 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 15 v 108 (ObjNum 131 snap 26 seq_num 131) dirty exists 2020-03-09T00:25:38.018 INFO:tasks.rados.rados.0.smithi104.stdout:319: left oid 15 (ObjNum 131 snap 26 seq_num 131) 2020-03-09T00:25:38.018 INFO:tasks.rados.rados.0.smithi104.stdout:320: finishing write tid 7 to smithi10428818-47 2020-03-09T00:25:38.019 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 47 v 146 (ObjNum 132 snap 26 seq_num 132) dirty exists 2020-03-09T00:25:38.019 INFO:tasks.rados.rados.0.smithi104.stdout:320: left oid 47 (ObjNum 132 snap 26 seq_num 132) 2020-03-09T00:25:38.019 INFO:tasks.rados.rados.0.smithi104.stdout:322: finishing rollback tid 0 to smithi10428818-40 2020-03-09T00:25:38.019 INFO:tasks.rados.rados.0.smithi104.stdout:322: finishing rollback tid 1 to smithi10428818-40 2020-03-09T00:25:38.019 INFO:tasks.rados.rados.0.smithi104.stdout:322: finishing rollback tid 2 to smithi10428818-40 2020-03-09T00:25:38.019 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 40 v 68 (ObjNum 104 snap 17 seq_num 104) dirty exists 2020-03-09T00:25:38.020 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:25:38.022 INFO:tasks.rados.rados.0.smithi104.stdout:319: done (3 left) 2020-03-09T00:25:38.022 INFO:tasks.rados.rados.0.smithi104.stdout:320: done (2 left) 2020-03-09T00:25:38.022 INFO:tasks.rados.rados.0.smithi104.stdout:322: done (1 left) 2020-03-09T00:25:38.022 INFO:tasks.rados.rados.0.smithi104.stdout:323: done (0 left) 2020-03-09T00:25:38.022 INFO:tasks.rados.rados.0.smithi104.stdout:324: read oid 44 snap -1 2020-03-09T00:25:38.022 INFO:tasks.rados.rados.0.smithi104.stdout:324: expect (ObjNum 116 snap 24 seq_num 116) 2020-03-09T00:25:38.022 INFO:tasks.rados.rados.0.smithi104.stdout:325: rollback oid 38 current snap is 26 2020-03-09T00:25:38.023 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 38 to 25 2020-03-09T00:25:38.023 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 45 current snap is 26 2020-03-09T00:25:38.023 INFO:tasks.rados.rados.0.smithi104.stdout:326: seq_num 134 ranges {753664=704512,1458176=475136,1933312=573440,2506752=507904} 2020-03-09T00:25:38.030 INFO:tasks.rados.rados.0.smithi104.stdout:326: writing smithi10428818-45 from 753664 to 1458176 tid 1 2020-03-09T00:25:38.033 INFO:tasks.rados.rados.0.smithi104.stdout:326: writing smithi10428818-45 from 1458176 to 1933312 tid 2 2020-03-09T00:25:38.036 INFO:tasks.rados.rados.0.smithi104.stdout:326: writing smithi10428818-45 from 1933312 to 2506752 tid 3 2020-03-09T00:25:38.040 INFO:tasks.rados.rados.0.smithi104.stdout:326: writing smithi10428818-45 from 2506752 to 3014656 tid 4 2020-03-09T00:25:38.040 INFO:tasks.rados.rados.0.smithi104.stdout:327: read oid 30 snap -1 2020-03-09T00:25:38.041 INFO:tasks.rados.rados.0.smithi104.stdout:327: expect deleted 2020-03-09T00:25:38.041 INFO:tasks.rados.rados.0.smithi104.stdout:328: read oid 13 snap -1 2020-03-09T00:25:38.041 INFO:tasks.rados.rados.0.smithi104.stdout:328: expect deleted 2020-03-09T00:25:38.041 INFO:tasks.rados.rados.0.smithi104.stdout:329: rollback oid 21 current snap is 26 2020-03-09T00:25:38.041 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 21 to 25 2020-03-09T00:25:38.041 INFO:tasks.rados.rados.0.smithi104.stdout:324: expect (ObjNum 116 snap 24 seq_num 116) 2020-03-09T00:25:38.065 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:38.061+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:38.122 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:38.125 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:38.124+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:38.126 INFO:tasks.rados.rados.0.smithi104.stdout:325: finishing rollback tid 1 to smithi10428818-38 2020-03-09T00:25:38.126 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 38 v 52 (ObjNum 0 snap 0 seq_num 0) dirty dne 2020-03-09T00:25:38.127 INFO:tasks.rados.rados.0.smithi104.stdout:324: done (5 left) 2020-03-09T00:25:38.127 INFO:tasks.rados.rados.0.smithi104.stdout:325: done (4 left) 2020-03-09T00:25:38.127 INFO:tasks.rados.rados.0.smithi104.stdout:330: read oid 49 snap 25 2020-03-09T00:25:38.127 INFO:tasks.rados.rados.0.smithi104.stdout:330: expect (ObjNum 42 snap 0 seq_num 42) 2020-03-09T00:25:38.127 INFO:tasks.rados.rados.0.smithi104.stdout:331: setattr oid 12 current snap is 26 2020-03-09T00:25:38.127 INFO:tasks.rados.rados.0.smithi104.stdout:326: finishing write tid 1 to smithi10428818-45 2020-03-09T00:25:38.127 INFO:tasks.rados.rados.0.smithi104.stdout:329: finishing rollback tid 0 to smithi10428818-21 2020-03-09T00:25:38.128 INFO:tasks.rados.rados.0.smithi104.stdout:329: finishing rollback tid 1 to smithi10428818-21 2020-03-09T00:25:38.128 INFO:tasks.rados.rados.0.smithi104.stdout:329: finishing rollback tid 2 to smithi10428818-21 2020-03-09T00:25:38.128 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 21 v 54 (ObjNum 110 snap 21 seq_num 110) dirty exists 2020-03-09T00:25:38.128 INFO:tasks.rados.rados.0.smithi104.stdout:326: finishing write tid 2 to smithi10428818-45 2020-03-09T00:25:38.128 INFO:tasks.rados.rados.0.smithi104.stdout:326: finishing write tid 3 to smithi10428818-45 2020-03-09T00:25:38.128 INFO:tasks.rados.rados.0.smithi104.stdout:326: finishing write tid 4 to smithi10428818-45 2020-03-09T00:25:38.128 INFO:tasks.rados.rados.0.smithi104.stdout:326: finishing write tid 6 to smithi10428818-45 2020-03-09T00:25:38.129 INFO:tasks.rados.rados.0.smithi104.stdout:326: finishing write tid 7 to smithi10428818-45 2020-03-09T00:25:38.129 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 45 v 60 (ObjNum 134 snap 26 seq_num 134) dirty exists 2020-03-09T00:25:38.129 INFO:tasks.rados.rados.0.smithi104.stdout:326: left oid 45 (ObjNum 134 snap 26 seq_num 134) 2020-03-09T00:25:38.129 INFO:tasks.rados.rados.0.smithi104.stdout:326: done (5 left) 2020-03-09T00:25:38.130 INFO:tasks.rados.rados.0.smithi104.stdout:327: done (4 left) 2020-03-09T00:25:38.130 INFO:tasks.rados.rados.0.smithi104.stdout:328: done (3 left) 2020-03-09T00:25:38.130 INFO:tasks.rados.rados.0.smithi104.stdout:329: done (2 left) 2020-03-09T00:25:38.130 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 35 current snap is 26 2020-03-09T00:25:38.130 INFO:tasks.rados.rados.0.smithi104.stdout:332: seq_num 135 ranges {393216=720896,1114112=622592,1736704=688128,2424832=245760} 2020-03-09T00:25:38.133 INFO:tasks.rados.rados.0.smithi104.stdout:332: writing smithi10428818-35 from 393216 to 1114112 tid 1 2020-03-09T00:25:38.137 INFO:tasks.rados.rados.0.smithi104.stdout:332: writing smithi10428818-35 from 1114112 to 1736704 tid 2 2020-03-09T00:25:38.141 INFO:tasks.rados.rados.0.smithi104.stdout:332: writing smithi10428818-35 from 1736704 to 2424832 tid 3 2020-03-09T00:25:38.144 INFO:tasks.rados.rados.0.smithi104.stdout:332: writing smithi10428818-35 from 2424832 to 2670592 tid 4 2020-03-09T00:25:38.144 INFO:tasks.rados.rados.0.smithi104.stdout:333: rmattr oid 9 current snap is 26 2020-03-09T00:25:38.144 INFO:tasks.rados.rados.0.smithi104.stdout:333: done (3 left) 2020-03-09T00:25:38.144 INFO:tasks.rados.rados.0.smithi104.stdout:334: copy_from oid 23 from oid 45 current snap is 26 2020-03-09T00:25:38.144 INFO:tasks.rados.rados.0.smithi104.stdout:330: expect (ObjNum 42 snap 0 seq_num 42) 2020-03-09T00:25:38.163 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:38.163 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:38.173 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:25:38.178 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 12 v 75 (ObjNum 75 snap 11 seq_num 75) dirty exists 2020-03-09T00:25:38.178 INFO:tasks.rados.rados.0.smithi104.stdout:330: done (3 left) 2020-03-09T00:25:38.178 INFO:tasks.rados.rados.0.smithi104.stdout:331: done (2 left) 2020-03-09T00:25:38.178 INFO:tasks.rados.rados.0.smithi104.stdout:335: rmattr oid 40 current snap is 26 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:332: finishing write tid 1 to smithi10428818-35 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:332: finishing write tid 2 to smithi10428818-35 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:332: finishing write tid 3 to smithi10428818-35 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:332: finishing write tid 4 to smithi10428818-35 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:332: finishing write tid 6 to smithi10428818-35 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:332: finishing write tid 7 to smithi10428818-35 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 35 v 92 (ObjNum 135 snap 26 seq_num 135) dirty exists 2020-03-09T00:25:38.179 INFO:tasks.rados.rados.0.smithi104.stdout:332: left oid 35 (ObjNum 135 snap 26 seq_num 135) 2020-03-09T00:25:38.180 INFO:tasks.rados.rados.0.smithi104.stdout:332: done (2 left) 2020-03-09T00:25:38.180 INFO:tasks.rados.rados.0.smithi104.stdout:336: copy_from oid 5 from oid 26 current snap is 26 2020-03-09T00:25:38.180 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 21 current snap is 26 2020-03-09T00:25:38.180 INFO:tasks.rados.rados.0.smithi104.stdout:337: seq_num 136 ranges {3620864=737280,4358144=770048,5128192=540672,5668864=245760} 2020-03-09T00:25:38.200 INFO:tasks.rados.rados.0.smithi104.stdout:337: writing smithi10428818-21 from 3620864 to 4358144 tid 1 2020-03-09T00:25:38.204 INFO:tasks.rados.rados.0.smithi104.stdout:337: writing smithi10428818-21 from 4358144 to 5128192 tid 2 2020-03-09T00:25:38.209 INFO:tasks.rados.rados.0.smithi104.stdout:337: writing smithi10428818-21 from 5128192 to 5668864 tid 3 2020-03-09T00:25:38.210 INFO:tasks.rados.rados.0.smithi104.stdout:337: writing smithi10428818-21 from 5668864 to 5914624 tid 4 2020-03-09T00:25:38.210 INFO:tasks.rados.rados.0.smithi104.stdout:338: rmattr oid 43 current snap is 26 2020-03-09T00:25:38.211 INFO:tasks.rados.rados.0.smithi104.stdout:338: done (4 left) 2020-03-09T00:25:38.211 INFO:tasks.rados.rados.0.smithi104.stdout:339: read oid 6 snap 25 2020-03-09T00:25:38.211 INFO:tasks.rados.rados.0.smithi104.stdout:339: expect deleted 2020-03-09T00:25:38.211 INFO:tasks.rados.rados.0.smithi104.stdout:340: rmattr oid 25 current snap is 26 2020-03-09T00:25:38.211 INFO:tasks.rados.rados.0.smithi104.stdout:340: done (5 left) 2020-03-09T00:25:38.211 INFO:tasks.rados.rados.0.smithi104.stdout:341: read oid 2 snap -1 2020-03-09T00:25:38.211 INFO:tasks.rados.rados.0.smithi104.stdout:341: expect (ObjNum 1 snap 0 seq_num 1) 2020-03-09T00:25:38.212 INFO:tasks.rados.rados.0.smithi104.stdout:342: snap_create 2020-03-09T00:25:38.212 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 40 v 69 (ObjNum 104 snap 17 seq_num 104) dirty exists 2020-03-09T00:25:38.212 INFO:tasks.rados.rados.0.smithi104.stdout:334: finishing copy_from to smithi10428818-23 2020-03-09T00:25:38.212 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 23 v 60 (ObjNum 134 snap 26 seq_num 134) dirty exists 2020-03-09T00:25:38.212 INFO:tasks.rados.rados.0.smithi104.stdout:334: finishing copy_from racing read to smithi10428818-23 2020-03-09T00:25:38.212 INFO:tasks.rados.rados.0.smithi104.stdout:336: finishing copy_from to smithi10428818-5 2020-03-09T00:25:38.212 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 5 v 112 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-03-09T00:25:38.213 INFO:tasks.rados.rados.0.smithi104.stdout:336: finishing copy_from racing read to smithi10428818-5 2020-03-09T00:25:38.214 INFO:tasks.rados.rados.0.smithi104.stdout:341: expect (ObjNum 1 snap 0 seq_num 1) 2020-03-09T00:25:38.220 INFO:tasks.rados.rados.0.smithi104.stdout:337: finishing write tid 1 to smithi10428818-21 2020-03-09T00:25:38.220 INFO:tasks.rados.rados.0.smithi104.stdout:337: finishing write tid 2 to smithi10428818-21 2020-03-09T00:25:38.220 INFO:tasks.rados.rados.0.smithi104.stdout:337: finishing write tid 3 to smithi10428818-21 2020-03-09T00:25:38.221 INFO:tasks.rados.rados.0.smithi104.stdout:337: finishing write tid 4 to smithi10428818-21 2020-03-09T00:25:38.221 INFO:tasks.rados.rados.0.smithi104.stdout:337: finishing write tid 6 to smithi10428818-21 2020-03-09T00:25:38.222 INFO:tasks.rados.rados.0.smithi104.stdout:337: finishing write tid 7 to smithi10428818-21 2020-03-09T00:25:38.222 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 21 v 59 (ObjNum 136 snap 26 seq_num 136) dirty exists 2020-03-09T00:25:38.222 INFO:tasks.rados.rados.0.smithi104.stdout:337: left oid 21 (ObjNum 136 snap 26 seq_num 136) 2020-03-09T00:25:38.226 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:38.224+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:38.275 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:38.296 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:38.301 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:38.302 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:38.311 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:25:38.316 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:38.327 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:38.326+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:38.336 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:38.356 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:38.376 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:38.396 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:38.416 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:38.428 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:38.426+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:38.437 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:38.463 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:38.468 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:38.468 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:38.468 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:38.468 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:38.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.469 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:38.469 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:38.469 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 50.638275047999997, 2020-03-09T00:25:38.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737, 2020-03-09T00:25:38.469 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.469 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.469 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:38.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000156164 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827665+0000", 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5397000000000001e-05 2020-03-09T00:25:38.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827690+0000", 2020-03-09T00:25:38.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6900000000000002e-05 2020-03-09T00:25:38.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827727+0000", 2020-03-09T00:25:38.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9529999999999999e-06 2020-03-09T00:25:38.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827735+0000", 2020-03-09T00:25:38.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011533660000000001 2020-03-09T00:25:38.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828889+0000", 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1542999999999999e-05 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828900+0000", 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038471730000000002 2020-03-09T00:25:38.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832747+0000", 2020-03-09T00:25:38.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8874000000000001e-05 2020-03-09T00:25:38.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.477 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832786+0000", 2020-03-09T00:25:38.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737 2020-03-09T00:25:38.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.477 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.477 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 38/32 ECSubWrite(tid=5, reqid=client.4751.0:207, at_version=38'4, trim_to=0'0, roll_forward_to=35'2))", 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 50.601677619999997, 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045310819999999996, 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.478 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:38.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864106+0000", 2020-03-09T00:25:38.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000157849 2020-03-09T00:25:38.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864264+0000", 2020-03-09T00:25:38.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8293000000000002e-05 2020-03-09T00:25:38.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.487 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864292+0000", 2020-03-09T00:25:38.487 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6464e-05 2020-03-09T00:25:38.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.487 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864309+0000", 2020-03-09T00:25:38.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9360000000000002e-06 2020-03-09T00:25:38.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.488 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.488 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.488 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864316+0000", 2020-03-09T00:25:38.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3850000000000001e-05 2020-03-09T00:25:38.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864370+0000", 2020-03-09T00:25:38.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3759e-05 2020-03-09T00:25:38.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.864383+0000", 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041983749999999999 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868582+0000", 2020-03-09T00:25:38.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5556000000000001e-05 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868637+0000", 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045310819999999996 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.491 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s1 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.672996654999999, 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997, 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.492 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062438599999999999 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793412+0000", 2020-03-09T00:25:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6639e-05 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793438+0000", 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3116999999999999e-05 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793461+0000", 2020-03-09T00:25:38.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2469999999999997e-06 2020-03-09T00:25:38.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793471+0000", 2020-03-09T00:25:38.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9004999999999994e-05 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793540+0000", 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8408000000000001e-05 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.497 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793558+0000", 2020-03-09T00:25:38.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053623589999999997 2020-03-09T00:25:38.498 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.498 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.498 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.498 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798920+0000", 2020-03-09T00:25:38.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2633000000000002e-05 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798973+0000", 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.499 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.500 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=30, reqid=client.4751.0:346, at_version=53'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:38.500 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:38.500 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.638738592999999, 2020-03-09T00:25:38.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999, 2020-03-09T00:25:38.500 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.501 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.501 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:38.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:38.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013296799999999999 2020-03-09T00:25:38.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827178+0000", 2020-03-09T00:25:38.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6305e-05 2020-03-09T00:25:38.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827205+0000", 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5832999999999999e-05 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827220+0000", 2020-03-09T00:25:38.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7170000000000008e-06 2020-03-09T00:25:38.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.523 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.523 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827228+0000", 2020-03-09T00:25:38.523 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2020-03-09T00:25:38.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827292+0000", 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4157e-05 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.524 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827306+0000", 2020-03-09T00:25:38.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046893439999999998 2020-03-09T00:25:38.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831995+0000", 2020-03-09T00:25:38.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3732000000000006e-05 2020-03-09T00:25:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832059+0000", 2020-03-09T00:25:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999 2020-03-09T00:25:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=31, reqid=client.4751.0:347, at_version=53'19, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.630900453000002, 2020-03-09T00:25:38.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996, 2020-03-09T00:25:38.529 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.529 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.529 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:38.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:38.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030527400000000002 2020-03-09T00:25:38.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835189+0000", 2020-03-09T00:25:38.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4273999999999998e-05 2020-03-09T00:25:38.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835213+0000", 2020-03-09T00:25:38.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7130999999999999e-05 2020-03-09T00:25:38.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.533 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835230+0000", 2020-03-09T00:25:38.533 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6400000000000003e-06 2020-03-09T00:25:38.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.533 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835239+0000", 2020-03-09T00:25:38.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0665000000000003e-05 2020-03-09T00:25:38.534 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.534 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.534 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.534 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835299+0000", 2020-03-09T00:25:38.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6062e-05 2020-03-09T00:25:38.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835315+0000", 2020-03-09T00:25:38.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005161349 2020-03-09T00:25:38.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840477+0000", 2020-03-09T00:25:38.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7466000000000003e-05 2020-03-09T00:25:38.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840524+0000", 2020-03-09T00:25:38.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996 2020-03-09T00:25:38.537 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.537 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.537 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.538 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=34, reqid=client.4751.0:349, at_version=0'0, trim_to=0'0, roll_forward_to=53'21))", 2020-03-09T00:25:38.538 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:38.538 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.620322403000003, 2020-03-09T00:25:38.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001, 2020-03-09T00:25:38.538 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.539 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.539 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:38.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:38.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6162000000000002e-05 2020-03-09T00:25:38.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845518+0000", 2020-03-09T00:25:38.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3681e-05 2020-03-09T00:25:38.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845531+0000", 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376e-05 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845547+0000", 2020-03-09T00:25:38.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8040000000000002e-06 2020-03-09T00:25:38.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845555+0000", 2020-03-09T00:25:38.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010135 2020-03-09T00:25:38.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845656+0000", 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2714000000000001e-05 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845669+0000", 2020-03-09T00:25:38.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072973140000000001 2020-03-09T00:25:38.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.545 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.545 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.852966+0000", 2020-03-09T00:25:38.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2704999999999998e-05 2020-03-09T00:25:38.545 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.545 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.546 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.853019+0000", 2020-03-09T00:25:38.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001 2020-03-09T00:25:38.546 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.546 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.546 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.547 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:38.547 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:38.547 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 29.551803565, 2020-03-09T00:25:38.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014134343000000001, 2020-03-09T00:25:38.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.547 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:38.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00044913599999999999 2020-03-09T00:25:38.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914429+0000", 2020-03-09T00:25:38.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3470000000000002e-05 2020-03-09T00:25:38.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914473+0000", 2020-03-09T00:25:38.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4422000000000001e-05 2020-03-09T00:25:38.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914497+0000", 2020-03-09T00:25:38.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4169999999999997e-06 2020-03-09T00:25:38.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914504+0000", 2020-03-09T00:25:38.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3522999999999995e-05 2020-03-09T00:25:38.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914567+0000", 2020-03-09T00:25:38.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4911e-05 2020-03-09T00:25:38.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914582+0000", 2020-03-09T00:25:38.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013495323 2020-03-09T00:25:38.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.928078+0000", 2020-03-09T00:25:38.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7141000000000002e-05 2020-03-09T00:25:38.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.928115+0000", 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014134343000000001 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.556 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 63/32 ECSubWrite(tid=83, reqid=osd.10.0:83, at_version=63'49, trim_to=0'0, roll_forward_to=60'47))", 2020-03-09T00:25:38.556 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:38.556 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.690382985999999, 2020-03-09T00:25:38.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073598240000000001, 2020-03-09T00:25:38.556 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.556 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.557 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:38.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:38.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4988000000000003e-05 2020-03-09T00:25:38.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775456+0000", 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6618000000000002e-05 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775483+0000", 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5108e-05 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775498+0000", 2020-03-09T00:25:38.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7239999999999992e-06 2020-03-09T00:25:38.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775505+0000", 2020-03-09T00:25:38.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110979 2020-03-09T00:25:38.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775616+0000", 2020-03-09T00:25:38.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2696e-05 2020-03-09T00:25:38.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775629+0000", 2020-03-09T00:25:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00708429 2020-03-09T00:25:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782713+0000", 2020-03-09T00:25:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7420999999999999e-05 2020-03-09T00:25:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782761+0000", 2020-03-09T00:25:38.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073598240000000001 2020-03-09T00:25:38.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.564 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.564 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.565 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 63/48 ECSubWrite(tid=42, reqid=osd.15.0:42, at_version=63'29, trim_to=0'0, roll_forward_to=59'27))", 2020-03-09T00:25:38.565 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:38.565 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.689610164000001, 2020-03-09T00:25:38.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005556643, 2020-03-09T00:25:38.565 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.566 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.566 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:38.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:38.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8814000000000003e-05 2020-03-09T00:25:38.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776233+0000", 2020-03-09T00:25:38.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2234000000000001e-05 2020-03-09T00:25:38.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776255+0000", 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6149000000000001e-05 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776271+0000", 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0389999999999999e-06 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776278+0000", 2020-03-09T00:25:38.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040392100000000001 2020-03-09T00:25:38.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776682+0000", 2020-03-09T00:25:38.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3020000000000001e-05 2020-03-09T00:25:38.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776695+0000", 2020-03-09T00:25:38.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049768449999999997 2020-03-09T00:25:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781672+0000", 2020-03-09T00:25:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8621e-05 2020-03-09T00:25:38.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781730+0000", 2020-03-09T00:25:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005556643 2020-03-09T00:25:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.573 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 63/32 ECSubWrite(tid=84, reqid=osd.10.0:84, at_version=63'51, trim_to=0'0, roll_forward_to=60'47))", 2020-03-09T00:25:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.689133607999999, 2020-03-09T00:25:38.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061568409999999997, 2020-03-09T00:25:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2492999999999999e-05 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776723+0000", 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2741000000000001e-05 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776746+0000", 2020-03-09T00:25:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5485000000000002e-05 2020-03-09T00:25:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776761+0000", 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4679999999999997e-06 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776767+0000", 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012315000000000001 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776891+0000", 2020-03-09T00:25:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2254e-05 2020-03-09T00:25:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776903+0000", 2020-03-09T00:25:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058656510000000004 2020-03-09T00:25:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782769+0000", 2020-03-09T00:25:38.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8599000000000001e-05 2020-03-09T00:25:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782807+0000", 2020-03-09T00:25:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061568409999999997 2020-03-09T00:25:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 63/32 ECSubWrite(tid=14, reqid=osd.3.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=63'12))", 2020-03-09T00:25:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.777416+0000", 2020-03-09T00:25:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.688367872000001, 2020-03-09T00:25:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044778049999999996, 2020-03-09T00:25:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777416+0000", 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777416+0000", 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3637999999999997e-05 2020-03-09T00:25:38.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777470+0000", 2020-03-09T00:25:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1753000000000001e-05 2020-03-09T00:25:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777481+0000", 2020-03-09T00:25:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4347e-05 2020-03-09T00:25:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777496+0000", 2020-03-09T00:25:38.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7529999999999998e-06 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777502+0000", 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000196279 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777698+0000", 2020-03-09T00:25:38.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1268e-05 2020-03-09T00:25:38.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.777709+0000", 2020-03-09T00:25:38.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041370770000000003 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781846+0000", 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7689999999999999e-05 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781894+0000", 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044778049999999996 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.592 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s1 63/54 ECSubWrite(tid=31, reqid=osd.0.0:31, at_version=63'42, trim_to=0'0, roll_forward_to=60'38))", 2020-03-09T00:25:38.593 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.781663+0000", 2020-03-09T00:25:38.593 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.684120926999999, 2020-03-09T00:25:38.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043990009999999996, 2020-03-09T00:25:38.593 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.593 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.593 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781663+0000", 2020-03-09T00:25:38.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781663+0000", 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8446000000000003e-05 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781721+0000", 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.933e-05 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781741+0000", 2020-03-09T00:25:38.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3501e-05 2020-03-09T00:25:38.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781754+0000", 2020-03-09T00:25:38.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5519999999999999e-06 2020-03-09T00:25:38.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781762+0000", 2020-03-09T00:25:38.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00059762200000000002 2020-03-09T00:25:38.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782359+0000", 2020-03-09T00:25:38.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2493999999999999e-05 2020-03-09T00:25:38.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782372+0000", 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0036379670000000002 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.786010+0000", 2020-03-09T00:25:38.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2089000000000002e-05 2020-03-09T00:25:38.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.786062+0000", 2020-03-09T00:25:38.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043990009999999996 2020-03-09T00:25:38.600 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 68/67 ECSubWrite(tid=91, reqid=client.4751.0:580, at_version=68'59, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:38.601 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.498465875000001, 2020-03-09T00:25:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006533311, 2020-03-09T00:25:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:38.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:38.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089329700000000004 2020-03-09T00:25:38.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968211+0000", 2020-03-09T00:25:38.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4736000000000001e-05 2020-03-09T00:25:38.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968246+0000", 2020-03-09T00:25:38.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0851000000000001e-05 2020-03-09T00:25:38.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968267+0000", 2020-03-09T00:25:38.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1970000000000001e-05 2020-03-09T00:25:38.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968279+0000", 2020-03-09T00:25:38.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6097999999999998e-05 2020-03-09T00:25:38.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968325+0000", 2020-03-09T00:25:38.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2437e-05 2020-03-09T00:25:38.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968337+0000", 2020-03-09T00:25:38.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054777970000000004 2020-03-09T00:25:38.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.973815+0000", 2020-03-09T00:25:38.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6124999999999997e-05 2020-03-09T00:25:38.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.973851+0000", 2020-03-09T00:25:38.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006533311 2020-03-09T00:25:38.612 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.612 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.612 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.612 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 68/67 ECSubWrite(tid=101, reqid=client.4751.0:592, at_version=68'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:38.613 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:38.613 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.476712416000002, 2020-03-09T00:25:38.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01056028, 2020-03-09T00:25:38.613 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.613 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.613 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:38.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00055521499999999996 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989627+0000", 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4139999999999999e-05 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989651+0000", 2020-03-09T00:25:38.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5061e-05 2020-03-09T00:25:38.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989666+0000", 2020-03-09T00:25:38.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1540000000000002e-06 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989674+0000", 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9898000000000001e-05 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989734+0000", 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5279000000000001e-05 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989749+0000", 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098335769999999996 2020-03-09T00:25:38.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999583+0000", 2020-03-09T00:25:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8955999999999997e-05 2020-03-09T00:25:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999632+0000", 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01056028 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.620 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 68/67 ECSubWrite(tid=102, reqid=client.4751.0:593, at_version=68'24, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:38.621 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:38.621 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 20.472770423, 2020-03-09T00:25:38.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066457310000000002, 2020-03-09T00:25:38.621 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.621 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.621 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.622 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.622 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:38.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:38.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037136200000000003 2020-03-09T00:25:38.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993385+0000", 2020-03-09T00:25:38.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2940999999999999e-05 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993408+0000", 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376e-05 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993423+0000", 2020-03-09T00:25:38.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6939999999999995e-06 2020-03-09T00:25:38.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993431+0000", 2020-03-09T00:25:38.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9996e-05 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993481+0000", 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6351999999999999e-05 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993537+0000", 2020-03-09T00:25:38.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061008299999999998 2020-03-09T00:25:38.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999638+0000", 2020-03-09T00:25:38.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1180000000000001e-05 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999659+0000", 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066457310000000002 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.631 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.631 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.631 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 69/32 ECSubWrite(tid=20, reqid=client.4751.0:599, at_version=69'19, trim_to=0'0, roll_forward_to=67'16))", 2020-03-09T00:25:38.631 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:38.631 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.671045257999999, 2020-03-09T00:25:38.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056539340000000002, 2020-03-09T00:25:38.631 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.632 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.632 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:38.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1377000000000004e-05 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794820+0000", 2020-03-09T00:25:38.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3682000000000001e-05 2020-03-09T00:25:38.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794844+0000", 2020-03-09T00:25:38.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4918e-05 2020-03-09T00:25:38.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794859+0000", 2020-03-09T00:25:38.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4799999999999998e-06 2020-03-09T00:25:38.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794865+0000", 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016106930000000001 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796476+0000", 2020-03-09T00:25:38.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3885e-05 2020-03-09T00:25:38.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796490+0000", 2020-03-09T00:25:38.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003855261 2020-03-09T00:25:38.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800345+0000", 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7638e-05 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800393+0000", 2020-03-09T00:25:38.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056539340000000002 2020-03-09T00:25:38.639 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.639 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.639 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.639 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:29.821440+0000", 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.6443442560000001, 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097473000000000004, 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821440+0000", 2020-03-09T00:25:38.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821440+0000", 2020-03-09T00:25:38.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013057190000000001 2020-03-09T00:25:38.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822745+0000", 2020-03-09T00:25:38.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8926999999999999e-05 2020-03-09T00:25:38.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822774+0000", 2020-03-09T00:25:38.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6112000000000001e-05 2020-03-09T00:25:38.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822790+0000", 2020-03-09T00:25:38.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2632999999999999e-05 2020-03-09T00:25:38.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822803+0000", 2020-03-09T00:25:38.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5909999999999997e-05 2020-03-09T00:25:38.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822869+0000", 2020-03-09T00:25:38.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4355999999999999e-05 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822883+0000", 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082614209999999997 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.831145+0000", 2020-03-09T00:25:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2222000000000003e-05 2020-03-09T00:25:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.831187+0000", 2020-03-09T00:25:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097473000000000004 2020-03-09T00:25:38.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.648 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.648 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.648 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.as1 81/32 ECSubRead(tid=34, to_read={3:51058371:::smithi10428818-28:head=0,536576,0}, subchunks={3:51058371:::smithi10428818-28:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:25:38.648 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:30.815219+0000", 2020-03-09T00:25:38.649 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 7.6505647200000002, 2020-03-09T00:25:38.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055806220000000004, 2020-03-09T00:25:38.650 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.650 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.650 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815219+0000", 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815219+0000", 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6774999999999998e-05 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815276+0000", 2020-03-09T00:25:38.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3097e-05 2020-03-09T00:25:38.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815289+0000", 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7640000000000001e-05 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815307+0000", 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0458e-05 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815317+0000", 2020-03-09T00:25:38.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013303589999999999 2020-03-09T00:25:38.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.816648+0000", 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041522929999999996 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.820800+0000", 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055806220000000004 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.655 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.656 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.656 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 85/67 ECSubWrite(tid=120, reqid=client.4751.0:760, at_version=85'36, trim_to=0'0, roll_forward_to=82'35))", 2020-03-09T00:25:38.656 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:34.884449+0000", 2020-03-09T00:25:38.656 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.5813348340000002, 2020-03-09T00:25:38.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046319020000000002, 2020-03-09T00:25:38.657 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.657 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.657 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884449+0000", 2020-03-09T00:25:38.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884449+0000", 2020-03-09T00:25:38.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037579699999999998 2020-03-09T00:25:38.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884825+0000", 2020-03-09T00:25:38.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0894e-05 2020-03-09T00:25:38.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884846+0000", 2020-03-09T00:25:38.663 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1015999999999999e-05 2020-03-09T00:25:38.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.663 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884857+0000", 2020-03-09T00:25:38.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3180000000000007e-06 2020-03-09T00:25:38.664 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884865+0000", 2020-03-09T00:25:38.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9815000000000003e-05 2020-03-09T00:25:38.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884915+0000", 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2629000000000001e-05 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884928+0000", 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040991459999999997 2020-03-09T00:25:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.889027+0000", 2020-03-09T00:25:38.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4286999999999997e-05 2020-03-09T00:25:38.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.889081+0000", 2020-03-09T00:25:38.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046319020000000002 2020-03-09T00:25:38.668 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.668 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.668 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.669 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 85/67 ECSubWrite(tid=124, reqid=client.4751.0:762, at_version=85'38, trim_to=0'0, roll_forward_to=85'37))", 2020-03-09T00:25:38.669 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:34.896749+0000", 2020-03-09T00:25:38.669 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.5690346829999999, 2020-03-09T00:25:38.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045755600000000002, 2020-03-09T00:25:38.669 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:38.670 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:38.670 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:38.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.670 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:38.670 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.896749+0000", 2020-03-09T00:25:38.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:38.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.896749+0000", 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036312900000000001 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:38.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.897112+0000", 2020-03-09T00:25:38.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5863999999999999e-05 2020-03-09T00:25:38.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:38.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.897128+0000", 2020-03-09T00:25:38.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0463e-05 2020-03-09T00:25:38.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:38.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.897139+0000", 2020-03-09T00:25:38.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4989999999999996e-06 2020-03-09T00:25:38.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.897143+0000", 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2046999999999999e-05 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.897185+0000", 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3722e-05 2020-03-09T00:25:38.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:38.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.897199+0000", 2020-03-09T00:25:38.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0040862709999999998 2020-03-09T00:25:38.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:38.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.901285+0000", 2020-03-09T00:25:38.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9564999999999998e-05 2020-03-09T00:25:38.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:38.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:38.678 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:38.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.901325+0000", 2020-03-09T00:25:38.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045755600000000002 2020-03-09T00:25:38.679 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.679 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.679 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.679 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:38.679 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:38.680 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:38.682 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:38.682 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:38.682 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:38.682 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:38.683 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:38.683 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.683 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.683 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.683 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.683 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:38.685 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:38.686 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:38.686 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:38.680+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:38.781 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:38.780+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:38.814 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:38.815 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:38.815 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:38.815 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:38.826 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:38.839 INFO:tasks.rados.rados.0.smithi104.stdout:334: done (6 left) 2020-03-09T00:25:38.839 INFO:tasks.rados.rados.0.smithi104.stdout:335: done (5 left) 2020-03-09T00:25:38.839 INFO:tasks.rados.rados.0.smithi104.stdout:336: done (4 left) 2020-03-09T00:25:38.839 INFO:tasks.rados.rados.0.smithi104.stdout:337: done (3 left) 2020-03-09T00:25:38.840 INFO:tasks.rados.rados.0.smithi104.stdout:339: done (2 left) 2020-03-09T00:25:38.840 INFO:tasks.rados.rados.0.smithi104.stdout:341: done (1 left) 2020-03-09T00:25:38.840 INFO:tasks.rados.rados.0.smithi104.stdout:342: done (0 left) 2020-03-09T00:25:38.840 INFO:tasks.rados.rados.0.smithi104.stdout:343: snap_remove snap 25 2020-03-09T00:25:38.850 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-03-09T00:25:38.881 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:38.880+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:38.951 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:38.951 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:38.952 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:38.960 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:38.982 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:38.981+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:39.085 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:39.082+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:39.126 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es1 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 45.269008049, 2020-03-09T00:25:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010037961, 2020-03-09T00:25:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854123+0000", 2020-03-09T00:25:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040949200000000002 2020-03-09T00:25:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854532+0000", 2020-03-09T00:25:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1631000000000003e-05 2020-03-09T00:25:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854564+0000", 2020-03-09T00:25:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2872000000000001e-05 2020-03-09T00:25:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854587+0000", 2020-03-09T00:25:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0699999999999999e-05 2020-03-09T00:25:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854598+0000", 2020-03-09T00:25:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6437000000000001e-05 2020-03-09T00:25:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854664+0000", 2020-03-09T00:25:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6180999999999998e-05 2020-03-09T00:25:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.854700+0000", 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094130399999999993 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864113+0000", 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7608000000000002e-05 2020-03-09T00:25:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864161+0000", 2020-03-09T00:25:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010037961 2020-03-09T00:25:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.271792279000003, 2020-03-09T00:25:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649, 2020-03-09T00:25:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022148699999999999 2020-03-09T00:25:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853554+0000", 2020-03-09T00:25:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7487e-05 2020-03-09T00:25:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853571+0000", 2020-03-09T00:25:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1065e-05 2020-03-09T00:25:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853582+0000", 2020-03-09T00:25:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7259999999999999e-06 2020-03-09T00:25:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853588+0000", 2020-03-09T00:25:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6578e-05 2020-03-09T00:25:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853634+0000", 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1282e-05 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853646+0000", 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012718002000000001 2020-03-09T00:25:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866364+0000", 2020-03-09T00:25:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6638999999999999e-05 2020-03-09T00:25:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866400+0000", 2020-03-09T00:25:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649 2020-03-09T00:25:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:375 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.322344264999998, 2020-03-09T00:25:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317, 2020-03-09T00:25:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 375 2020-03-09T00:25:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001381363 2020-03-09T00:25:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802168+0000", 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5639000000000001e-05 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802184+0000", 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6752000000000001e-05 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802240+0000", 2020-03-09T00:25:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7064999999999999e-05 2020-03-09T00:25:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802268+0000", 2020-03-09T00:25:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6162999999999996e-05 2020-03-09T00:25:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802344+0000", 2020-03-09T00:25:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000166458 2020-03-09T00:25:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802510+0000", 2020-03-09T00:25:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018314410000000001 2020-03-09T00:25:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804342+0000", 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079462270000000002 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812288+0000", 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078060580000000003 2020-03-09T00:25:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820094+0000", 2020-03-09T00:25:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110151 2020-03-09T00:25:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820204+0000", 2020-03-09T00:25:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317 2020-03-09T00:25:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:376 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.318988930000003, 2020-03-09T00:25:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127, 2020-03-09T00:25:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 376 2020-03-09T00:25:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00133916 2020-03-09T00:25:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805481+0000", 2020-03-09T00:25:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1296e-05 2020-03-09T00:25:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805492+0000", 2020-03-09T00:25:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1435999999999999e-05 2020-03-09T00:25:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805514+0000", 2020-03-09T00:25:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5969999999999998e-06 2020-03-09T00:25:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805524+0000", 2020-03-09T00:25:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026245270000000002 2020-03-09T00:25:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808148+0000", 2020-03-09T00:25:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000157388 2020-03-09T00:25:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808305+0000", 2020-03-09T00:25:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00107099 2020-03-09T00:25:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.809376+0000", 2020-03-09T00:25:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071133840000000004 2020-03-09T00:25:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816490+0000", 2020-03-09T00:25:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038222479999999999 2020-03-09T00:25:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820312+0000", 2020-03-09T00:25:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012910100000000001 2020-03-09T00:25:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820441+0000", 2020-03-09T00:25:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127 2020-03-09T00:25:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:377 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~245760] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.315129517999999, 2020-03-09T00:25:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858, 2020-03-09T00:25:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 377 2020-03-09T00:25:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00047463399999999998 2020-03-09T00:25:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808476+0000", 2020-03-09T00:25:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7579999999999998e-06 2020-03-09T00:25:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808482+0000", 2020-03-09T00:25:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7729999999999994e-06 2020-03-09T00:25:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808492+0000", 2020-03-09T00:25:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808496+0000", 2020-03-09T00:25:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003930813 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812427+0000", 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013734399999999999 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812564+0000", 2020-03-09T00:25:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00094040499999999995 2020-03-09T00:25:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.813505+0000", 2020-03-09T00:25:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030313029999999999 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816536+0000", 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048022810000000003 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821338+0000", 2020-03-09T00:25:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4952999999999997e-05 2020-03-09T00:25:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821433+0000", 2020-03-09T00:25:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858 2020-03-09T00:25:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:378 3.2s0 3:482d6e06:::smithi10428818-17:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.314601394, 2020-03-09T00:25:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001, 2020-03-09T00:25:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 378 2020-03-09T00:25:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2327999999999999e-05 2020-03-09T00:25:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808572+0000", 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7400000000000002e-06 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808576+0000", 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5849999999999996e-06 2020-03-09T00:25:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808582+0000", 2020-03-09T00:25:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7070000000000001e-06 2020-03-09T00:25:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808585+0000", 2020-03-09T00:25:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068015719999999997 2020-03-09T00:25:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815386+0000", 2020-03-09T00:25:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124704 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815511+0000", 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00063270399999999999 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816144+0000", 2020-03-09T00:25:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056543100000000003 2020-03-09T00:25:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816709+0000", 2020-03-09T00:25:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048141440000000002 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821523+0000", 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012666000000000001 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821650+0000", 2020-03-09T00:25:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001 2020-03-09T00:25:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:379 3.2s0 3:482d6e06:::smithi10428818-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.314521374000002, 2020-03-09T00:25:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999, 2020-03-09T00:25:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 379 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3624000000000002e-05 2020-03-09T00:25:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808643+0000", 2020-03-09T00:25:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.625e-06 2020-03-09T00:25:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808647+0000", 2020-03-09T00:25:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1500000000000002e-06 2020-03-09T00:25:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808654+0000", 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8480000000000001e-06 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808657+0000", 2020-03-09T00:25:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079120670000000001 2020-03-09T00:25:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816569+0000", 2020-03-09T00:25:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012729000000000001 2020-03-09T00:25:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816696+0000", 2020-03-09T00:25:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054928110000000002 2020-03-09T00:25:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822189+0000", 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6662000000000005e-05 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822276+0000", 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001870029 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.824146+0000", 2020-03-09T00:25:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999 2020-03-09T00:25:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:503 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.833961+0000", 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.289170346999999, 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093609360000000003, 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 503 2020-03-09T00:25:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.833961+0000", 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.833961+0000", 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9160999999999999e-05 2020-03-09T00:25:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834010+0000", 2020-03-09T00:25:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0647e-05 2020-03-09T00:25:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834020+0000", 2020-03-09T00:25:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7700999999999999e-05 2020-03-09T00:25:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834038+0000", 2020-03-09T00:25:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1950000000000003e-06 2020-03-09T00:25:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834046+0000", 2020-03-09T00:25:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1390999999999997e-05 2020-03-09T00:25:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834088+0000", 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8152000000000006e-05 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834176+0000", 2020-03-09T00:25:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064286300000000003 2020-03-09T00:25:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834819+0000", 2020-03-09T00:25:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038976039999999998 2020-03-09T00:25:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838716+0000", 2020-03-09T00:25:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004520647 2020-03-09T00:25:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843237+0000", 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4574999999999996e-05 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843322+0000", 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0093609360000000003 2020-03-09T00:25:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:504 3.2s0 3:482d6e06:::smithi10428818-17:head [rollback 15] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.834092+0000", 2020-03-09T00:25:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.289038808000001, 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094498250000000002, 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 504 2020-03-09T00:25:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834092+0000", 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834092+0000", 2020-03-09T00:25:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6925999999999998e-05 2020-03-09T00:25:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834139+0000", 2020-03-09T00:25:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6800000000000004e-06 2020-03-09T00:25:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834146+0000", 2020-03-09T00:25:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2583e-05 2020-03-09T00:25:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834158+0000", 2020-03-09T00:25:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3390000000000003e-06 2020-03-09T00:25:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834163+0000", 2020-03-09T00:25:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00100004 2020-03-09T00:25:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.835163+0000", 2020-03-09T00:25:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011991 2020-03-09T00:25:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.835283+0000", 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092352899999999997 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.836206+0000", 2020-03-09T00:25:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025829730000000001 2020-03-09T00:25:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838789+0000", 2020-03-09T00:25:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046750439999999997 2020-03-09T00:25:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843464+0000", 2020-03-09T00:25:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7800999999999996e-05 2020-03-09T00:25:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.843542+0000", 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094498250000000002 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:505 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.288921950999999, 2020-03-09T00:25:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016627194000000001, 2020-03-09T00:25:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 505 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7997000000000002e-05 2020-03-09T00:25:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834247+0000", 2020-03-09T00:25:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000001e-06 2020-03-09T00:25:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834251+0000", 2020-03-09T00:25:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.469e-06 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834258+0000", 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2660000000000002e-06 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834262+0000", 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032379409999999998 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837500+0000", 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013429800000000001 2020-03-09T00:25:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837634+0000", 2020-03-09T00:25:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076062099999999995 2020-03-09T00:25:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838394+0000", 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039555900000000003 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838790+0000", 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01196025 2020-03-09T00:25:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.850750+0000", 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5890999999999995e-05 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.850836+0000", 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016627194000000001 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as3 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.914124+0000", 2020-03-09T00:25:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.209006810000002, 2020-03-09T00:25:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094380360000000003, 2020-03-09T00:25:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914124+0000", 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914124+0000", 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010665379999999999 2020-03-09T00:25:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915191+0000", 2020-03-09T00:25:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9760000000000001e-05 2020-03-09T00:25:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915210+0000", 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2631e-05 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915223+0000", 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.143e-06 2020-03-09T00:25:39.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915227+0000", 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9517e-05 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915287+0000", 2020-03-09T00:25:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8060000000000006e-06 2020-03-09T00:25:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915297+0000", 2020-03-09T00:25:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082301200000000005 2020-03-09T00:25:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923527+0000", 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5521000000000001e-05 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923562+0000", 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094380360000000003 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:587 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~589824] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.146980074999998, 2020-03-09T00:25:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015855035, 2020-03-09T00:25:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 587 2020-03-09T00:25:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001387634 2020-03-09T00:25:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977539+0000", 2020-03-09T00:25:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3190000000000001e-06 2020-03-09T00:25:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977548+0000", 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5709000000000001e-05 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977574+0000", 2020-03-09T00:25:39.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.471e-05 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977588+0000", 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8529000000000006e-05 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977677+0000", 2020-03-09T00:25:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013839599999999999 2020-03-09T00:25:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977815+0000", 2020-03-09T00:25:39.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015356079999999999 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979351+0000", 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070753639999999998 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986426+0000", 2020-03-09T00:25:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054801509999999999 2020-03-09T00:25:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991906+0000", 2020-03-09T00:25:39.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9615000000000001e-05 2020-03-09T00:25:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992006+0000", 2020-03-09T00:25:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015855035 2020-03-09T00:25:39.250 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:588 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~540672] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.143922387, 2020-03-09T00:25:39.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013267434999999999, 2020-03-09T00:25:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 588 2020-03-09T00:25:39.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001696822 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980905+0000", 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7710000000000006e-06 2020-03-09T00:25:39.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980913+0000", 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4919999999999999e-05 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980928+0000", 2020-03-09T00:25:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.34e-06 2020-03-09T00:25:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980935+0000", 2020-03-09T00:25:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015720580000000001 2020-03-09T00:25:39.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.982507+0000", 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7051000000000004e-05 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.982595+0000", 2020-03-09T00:25:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081195499999999995 2020-03-09T00:25:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983406+0000", 2020-03-09T00:25:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082914860000000007 2020-03-09T00:25:39.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991698+0000", 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058976000000000005 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992288+0000", 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018827199999999999 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992476+0000", 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013267434999999999 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:589 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~622592] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.139575484000002, 2020-03-09T00:25:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012556799, 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 589 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019622569999999998 2020-03-09T00:25:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985518+0000", 2020-03-09T00:25:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3129999999999999e-06 2020-03-09T00:25:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985526+0000", 2020-03-09T00:25:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4384e-05 2020-03-09T00:25:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985540+0000", 2020-03-09T00:25:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2789999999999998e-06 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985549+0000", 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097411399999999999 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986523+0000", 2020-03-09T00:25:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5084000000000001e-05 2020-03-09T00:25:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986598+0000", 2020-03-09T00:25:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010282659999999999 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987626+0000", 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040731630000000003 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991699+0000", 2020-03-09T00:25:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043120320000000004 2020-03-09T00:25:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996011+0000", 2020-03-09T00:25:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100907 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996112+0000", 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012556799 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:590 3.2s0 3:49883c8e:::smithi10428818-40:head [setxattr _header (57) in=7b] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.137522085000001, 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010789646, 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 590 2020-03-09T00:25:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2869999999999998e-05 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985692+0000", 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.507e-06 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985700+0000", 2020-03-09T00:25:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5087e-05 2020-03-09T00:25:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985715+0000", 2020-03-09T00:25:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5260000000000001e-06 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985721+0000", 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051320369999999999 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.990853+0000", 2020-03-09T00:25:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114726 2020-03-09T00:25:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.990968+0000", 2020-03-09T00:25:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000440037 2020-03-09T00:25:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991408+0000", 2020-03-09T00:25:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001238942 2020-03-09T00:25:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992647+0000", 2020-03-09T00:25:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036263749999999998 2020-03-09T00:25:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996273+0000", 2020-03-09T00:25:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012553900000000001 2020-03-09T00:25:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996399+0000", 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010789646 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:591 3.2s0 3:49883c8e:::smithi10428818-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.137362948, 2020-03-09T00:25:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015712046, 2020-03-09T00:25:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 591 2020-03-09T00:25:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3307999999999999e-05 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985821+0000", 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5400000000000001e-06 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985828+0000", 2020-03-09T00:25:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2712e-05 2020-03-09T00:25:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985841+0000", 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6920000000000001e-06 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985845+0000", 2020-03-09T00:25:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066716329999999997 2020-03-09T00:25:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992517+0000", 2020-03-09T00:25:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012270799999999999 2020-03-09T00:25:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992640+0000", 2020-03-09T00:25:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00472662 2020-03-09T00:25:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.997366+0000", 2020-03-09T00:25:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012992200000000001 2020-03-09T00:25:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.997496+0000", 2020-03-09T00:25:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039839109999999997 2020-03-09T00:25:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:18.001480+0000", 2020-03-09T00:25:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015712046 2020-03-09T00:25:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:667 3.2s0 3:482d6e06:::smithi10428818-17:head [delete] snapc 2b=[2b,2a,24,23] ondisk+write+peerstat_old+known_if_redirected e79)", 2020-03-09T00:25:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:28.797272+0000", 2020-03-09T00:25:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.325858824999999, 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078211610000000001, 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 667 2020-03-09T00:25:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797272+0000", 2020-03-09T00:25:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797272+0000", 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9586000000000003e-05 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797322+0000", 2020-03-09T00:25:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797329+0000", 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2908000000000001e-05 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797342+0000", 2020-03-09T00:25:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8410000000000006e-06 2020-03-09T00:25:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797350+0000", 2020-03-09T00:25:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6886999999999998e-05 2020-03-09T00:25:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797417+0000", 2020-03-09T00:25:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8030000000000003e-06 2020-03-09T00:25:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.797426+0000", 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018565400000000001 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.799283+0000", 2020-03-09T00:25:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116297 2020-03-09T00:25:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.799399+0000", 2020-03-09T00:25:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092098600000000005 2020-03-09T00:25:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.800320+0000", 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018654640000000001 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.802185+0000", 2020-03-09T00:25:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028045570000000001 2020-03-09T00:25:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.804990+0000", 2020-03-09T00:25:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010326400000000001 2020-03-09T00:25:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:28.805093+0000", 2020-03-09T00:25:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078211610000000001 2020-03-09T00:25:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s6 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:29.821736+0000", 2020-03-09T00:25:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.3013947510000001, 2020-03-09T00:25:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013523835, 2020-03-09T00:25:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821736+0000", 2020-03-09T00:25:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821736+0000", 2020-03-09T00:25:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011537120000000001 2020-03-09T00:25:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822890+0000", 2020-03-09T00:25:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3646000000000003e-05 2020-03-09T00:25:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822924+0000", 2020-03-09T00:25:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9941000000000001e-05 2020-03-09T00:25:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822943+0000", 2020-03-09T00:25:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0580000000000004e-06 2020-03-09T00:25:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822953+0000", 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4598999999999996e-05 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.823017+0000", 2020-03-09T00:25:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3725e-05 2020-03-09T00:25:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.823031+0000", 2020-03-09T00:25:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012191274 2020-03-09T00:25:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.835222+0000", 2020-03-09T00:25:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7880000000000003e-05 2020-03-09T00:25:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.835260+0000", 2020-03-09T00:25:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013523835 2020-03-09T00:25:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s2 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.885073+0000", 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.2380576799999998, 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015418995, 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.885073+0000", 2020-03-09T00:25:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.885073+0000", 2020-03-09T00:25:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027330150000000001 2020-03-09T00:25:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887806+0000", 2020-03-09T00:25:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1937999999999998e-05 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887838+0000", 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7982000000000001e-05 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887866+0000", 2020-03-09T00:25:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1049999999999997e-06 2020-03-09T00:25:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887873+0000", 2020-03-09T00:25:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7388e-05 2020-03-09T00:25:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887931+0000", 2020-03-09T00:25:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2594e-05 2020-03-09T00:25:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887943+0000", 2020-03-09T00:25:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01248904 2020-03-09T00:25:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900432+0000", 2020-03-09T00:25:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9932999999999997e-05 2020-03-09T00:25:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900492+0000", 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015418995 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s6 85/67 ECSubWrite(tid=124, reqid=client.4751.0:762, at_version=85'38, trim_to=0'0, roll_forward_to=85'37))", 2020-03-09T00:25:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.896535+0000", 2020-03-09T00:25:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.226595788, 2020-03-09T00:25:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078862959999999992, 2020-03-09T00:25:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896535+0000", 2020-03-09T00:25:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896535+0000", 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044163500000000001 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896977+0000", 2020-03-09T00:25:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7209000000000001e-05 2020-03-09T00:25:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896994+0000", 2020-03-09T00:25:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6622000000000001e-05 2020-03-09T00:25:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897011+0000", 2020-03-09T00:25:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0329999999999992e-06 2020-03-09T00:25:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897019+0000", 2020-03-09T00:25:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6662000000000001e-05 2020-03-09T00:25:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897075+0000", 2020-03-09T00:25:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6526000000000001e-05 2020-03-09T00:25:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897092+0000", 2020-03-09T00:25:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072858999999999997 2020-03-09T00:25:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.904378+0000", 2020-03-09T00:25:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3708999999999998e-05 2020-03-09T00:25:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.904421+0000", 2020-03-09T00:25:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078862959999999992 2020-03-09T00:25:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:39.335 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:39.336 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:39.335+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:39.336 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:25:39.437 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:39.436+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:39.453 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:39.454 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:39.454 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:39.454 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:39.462 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:25:39.538 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:39.536+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:39.618 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:39.618 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:39.618 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:39.619 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:39.619 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:39.627 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:25:39.672 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:39.671+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:39.739 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:39.737+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:39.746 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:39.746 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:39.746 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:39.746 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:39.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.747 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:242 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~638976] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:39.747 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:39.747 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.896525994000001, 2020-03-09T00:25:39.748 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001, 2020-03-09T00:25:39.748 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.748 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.748 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.749 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.749 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.749 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 242 2020-03-09T00:25:39.749 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.750 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.750 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.750 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.750 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:39.750 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.751 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.751 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.751 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.751 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:39.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012420980000000001 2020-03-09T00:25:39.752 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.752 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.752 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.753 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847650+0000", 2020-03-09T00:25:39.753 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2689999999999996e-06 2020-03-09T00:25:39.753 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.753 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.753 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.754 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847657+0000", 2020-03-09T00:25:39.754 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2094e-05 2020-03-09T00:25:39.754 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.754 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.755 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.755 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847669+0000", 2020-03-09T00:25:39.755 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2690000000000002e-06 2020-03-09T00:25:39.755 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.756 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.756 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847676+0000", 2020-03-09T00:25:39.756 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3765999999999999e-05 2020-03-09T00:25:39.757 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.757 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.757 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.757 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847740+0000", 2020-03-09T00:25:39.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000155716 2020-03-09T00:25:39.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.758 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.758 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847896+0000", 2020-03-09T00:25:39.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020523669999999998 2020-03-09T00:25:39.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.759 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849948+0000", 2020-03-09T00:25:39.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010549378999999999 2020-03-09T00:25:39.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.762 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.763 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.763 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860497+0000", 2020-03-09T00:25:39.763 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046388970000000003 2020-03-09T00:25:39.763 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.763 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.764 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.764 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865136+0000", 2020-03-09T00:25:39.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2514000000000005e-05 2020-03-09T00:25:39.764 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.765 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.765 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.765 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865199+0000", 2020-03-09T00:25:39.765 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001 2020-03-09T00:25:39.766 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.766 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.766 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.767 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:243 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~376832] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:39.767 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:39.767 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.894020191000003, 2020-03-09T00:25:39.768 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999, 2020-03-09T00:25:39.768 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.768 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.768 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.768 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.769 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.769 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 243 2020-03-09T00:25:39.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.769 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:39.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:39.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012192539999999999 2020-03-09T00:25:39.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850133+0000", 2020-03-09T00:25:39.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1730000000000001e-06 2020-03-09T00:25:39.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850143+0000", 2020-03-09T00:25:39.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1009000000000001e-05 2020-03-09T00:25:39.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850174+0000", 2020-03-09T00:25:39.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0500000000000003e-06 2020-03-09T00:25:39.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850181+0000", 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047873150000000003 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.854968+0000", 2020-03-09T00:25:39.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013640000000000001 2020-03-09T00:25:39.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855104+0000", 2020-03-09T00:25:39.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001096669 2020-03-09T00:25:39.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856201+0000", 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042977170000000004 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860499+0000", 2020-03-09T00:25:39.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.008534827 2020-03-09T00:25:39.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869034+0000", 2020-03-09T00:25:39.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4033e-05 2020-03-09T00:25:39.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869118+0000", 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.783 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:244 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (56) in=7b] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:39.783 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:39.783 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.892674972000002, 2020-03-09T00:25:39.783 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999, 2020-03-09T00:25:39.783 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.783 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.783 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 244 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.784 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000146107 2020-03-09T00:25:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850405+0000", 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2099999999999999e-05 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850418+0000", 2020-03-09T00:25:39.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9311000000000001e-05 2020-03-09T00:25:39.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850447+0000", 2020-03-09T00:25:39.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6460000000000004e-06 2020-03-09T00:25:39.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850453+0000", 2020-03-09T00:25:39.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011322660999999999 2020-03-09T00:25:39.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861776+0000", 2020-03-09T00:25:39.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015394900000000001 2020-03-09T00:25:39.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.790 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861930+0000", 2020-03-09T00:25:39.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076229599999999998 2020-03-09T00:25:39.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862692+0000", 2020-03-09T00:25:39.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002593092 2020-03-09T00:25:39.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865285+0000", 2020-03-09T00:25:39.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039673670000000003 2020-03-09T00:25:39.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869253+0000", 2020-03-09T00:25:39.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010661 2020-03-09T00:25:39.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.793 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869359+0000", 2020-03-09T00:25:39.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:245 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:39.794 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.892428737000003, 2020-03-09T00:25:39.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998, 2020-03-09T00:25:39.795 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.795 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:39.795 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.795 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.795 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.795 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 245 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:39.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2926999999999997e-05 2020-03-09T00:25:39.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.797 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850548+0000", 2020-03-09T00:25:39.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7159999999999997e-06 2020-03-09T00:25:39.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850555+0000", 2020-03-09T00:25:39.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2537999999999999e-05 2020-03-09T00:25:39.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850568+0000", 2020-03-09T00:25:39.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6200000000000004e-06 2020-03-09T00:25:39.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.801 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.801 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850573+0000", 2020-03-09T00:25:39.801 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013277631 2020-03-09T00:25:39.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863851+0000", 2020-03-09T00:25:39.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016975200000000001 2020-03-09T00:25:39.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:39.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864021+0000", 2020-03-09T00:25:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058362869999999999 2020-03-09T00:25:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869857+0000", 2020-03-09T00:25:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011197 2020-03-09T00:25:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869969+0000", 2020-03-09T00:25:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0016289570000000001 2020-03-09T00:25:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.871598+0000", 2020-03-09T00:25:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998 2020-03-09T00:25:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:289 3.es0 3:73be088e:::smithi10428818-42:head [append 0~737280] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.928456500999999, 2020-03-09T00:25:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001, 2020-03-09T00:25:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001180787 2020-03-09T00:25:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815659+0000", 2020-03-09T00:25:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5960000000000004e-06 2020-03-09T00:25:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815667+0000", 2020-03-09T00:25:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7425e-05 2020-03-09T00:25:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815685+0000", 2020-03-09T00:25:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0162e-05 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815695+0000", 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2490000000000004e-05 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815787+0000", 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7851999999999998e-05 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815885+0000", 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001491497 2020-03-09T00:25:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817377+0000", 2020-03-09T00:25:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010162513999999999 2020-03-09T00:25:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827539+0000", 2020-03-09T00:25:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027651400000000001 2020-03-09T00:25:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830304+0000", 2020-03-09T00:25:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8808999999999995e-05 2020-03-09T00:25:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830403+0000", 2020-03-09T00:25:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001 2020-03-09T00:25:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:292 3.es0 3:73be088e:::smithi10428818-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.925153477999999, 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999, 2020-03-09T00:25:39.816 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.817 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:39.817 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.817 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.817 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.817 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 292 2020-03-09T00:25:39.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.818 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:39.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7518000000000001e-05 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817828+0000", 2020-03-09T00:25:39.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:39.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817834+0000", 2020-03-09T00:25:39.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2721000000000001e-05 2020-03-09T00:25:39.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817847+0000", 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2339999999999996e-06 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817851+0000", 2020-03-09T00:25:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012178102999999999 2020-03-09T00:25:39.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.822 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830029+0000", 2020-03-09T00:25:39.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012795100000000001 2020-03-09T00:25:39.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830157+0000", 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003501414 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833659+0000", 2020-03-09T00:25:39.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5081999999999996e-05 2020-03-09T00:25:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833744+0000", 2020-03-09T00:25:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002393077 2020-03-09T00:25:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.836137+0000", 2020-03-09T00:25:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999 2020-03-09T00:25:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:339 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 36.935245907000002, 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998, 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.826 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.827 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.827 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.827 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 339 2020-03-09T00:25:39.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.827 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00074630200000000001 2020-03-09T00:25:39.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.829 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.829 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808435+0000", 2020-03-09T00:25:39.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6087e-05 2020-03-09T00:25:39.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808451+0000", 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.413e-05 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808465+0000", 2020-03-09T00:25:39.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4320000000000004e-06 2020-03-09T00:25:39.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808471+0000", 2020-03-09T00:25:39.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046408889999999996 2020-03-09T00:25:39.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813112+0000", 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013929799999999999 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813252+0000", 2020-03-09T00:25:39.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017257030000000001 2020-03-09T00:25:39.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.814977+0000", 2020-03-09T00:25:39.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099845400000000001 2020-03-09T00:25:39.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.824962+0000", 2020-03-09T00:25:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049262399999999999 2020-03-09T00:25:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825454+0000", 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.462e-05 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825539+0000", 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998 2020-03-09T00:25:39.836 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.837 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.837 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.837 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:340 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~294912] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:39.837 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:39.837 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 36.932641478999997, 2020-03-09T00:25:39.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244, 2020-03-09T00:25:39.838 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.838 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.838 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.838 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.838 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.838 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 340 2020-03-09T00:25:39.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.839 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:39.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000216388 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.840 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810509+0000", 2020-03-09T00:25:39.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6459999999999998e-06 2020-03-09T00:25:39.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.841 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.841 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810515+0000", 2020-03-09T00:25:39.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.799e-06 2020-03-09T00:25:39.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810524+0000", 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.842 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810528+0000", 2020-03-09T00:25:39.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007805228 2020-03-09T00:25:39.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818333+0000", 2020-03-09T00:25:39.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013953500000000001 2020-03-09T00:25:39.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818473+0000", 2020-03-09T00:25:39.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00094954999999999996 2020-03-09T00:25:39.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.819423+0000", 2020-03-09T00:25:39.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056089679999999998 2020-03-09T00:25:39.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825032+0000", 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015242280000000001 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826556+0000", 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4308000000000007e-05 2020-03-09T00:25:39.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.848 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826640+0000", 2020-03-09T00:25:39.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244 2020-03-09T00:25:39.848 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.848 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.848 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.849 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.849 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.849 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:341 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:39.849 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:39.849 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 36.932376396000002, 2020-03-09T00:25:39.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001, 2020-03-09T00:25:39.849 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 341 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.850 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6538999999999997e-05 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810594+0000", 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3629999999999998e-06 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.853 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.853 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810598+0000", 2020-03-09T00:25:39.853 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1715e-05 2020-03-09T00:25:39.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.853 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.853 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810610+0000", 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7019999999999999e-06 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810613+0000", 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016583064000000002 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.854 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.855 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.855 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827196+0000", 2020-03-09T00:25:39.855 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000134802 2020-03-09T00:25:39.855 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.855 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.855 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827331+0000", 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079633999999999998 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828127+0000", 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000914331 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.856 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.857 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.857 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.829042+0000", 2020-03-09T00:25:39.857 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045226199999999998 2020-03-09T00:25:39.857 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.857 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.857 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.857 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000101851 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833666+0000", 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.858 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.859 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.859 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.859 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.859 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:342 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:39.860 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:39.860 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 36.932298295000002, 2020-03-09T00:25:39.860 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001, 2020-03-09T00:25:39.860 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.860 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:39.860 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.860 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.861 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.861 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 342 2020-03-09T00:25:39.861 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.861 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.861 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.862 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.862 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:39.862 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.862 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.862 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.862 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.863 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:39.863 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1574e-05 2020-03-09T00:25:39.863 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.863 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.863 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.863 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810658+0000", 2020-03-09T00:25:39.863 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.196e-06 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810661+0000", 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6000000000000003e-06 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.864 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810667+0000", 2020-03-09T00:25:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3269999999999999e-06 2020-03-09T00:25:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810670+0000", 2020-03-09T00:25:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018222015000000001 2020-03-09T00:25:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828892+0000", 2020-03-09T00:25:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107623 2020-03-09T00:25:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828999+0000", 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005589938 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834589+0000", 2020-03-09T00:25:39.867 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012716299999999999 2020-03-09T00:25:39.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.868 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.868 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.868 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834716+0000", 2020-03-09T00:25:39.868 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048828580000000003 2020-03-09T00:25:39.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.868 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.839599+0000", 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.870 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:374 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:39.870 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:39.870 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 35.948791472000003, 2020-03-09T00:25:39.870 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001, 2020-03-09T00:25:39.870 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.870 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:39.870 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.871 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.871 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.871 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 374 2020-03-09T00:25:39.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.871 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3215999999999998e-05 2020-03-09T00:25:39.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.873 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794216+0000", 2020-03-09T00:25:39.873 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7970000000000003e-06 2020-03-09T00:25:39.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794223+0000", 2020-03-09T00:25:39.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2177e-05 2020-03-09T00:25:39.874 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.874 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794235+0000", 2020-03-09T00:25:39.875 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8439999999999997e-06 2020-03-09T00:25:39.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.875 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.875 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.875 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794240+0000", 2020-03-09T00:25:39.875 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7449999999999997e-05 2020-03-09T00:25:39.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.876 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.876 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.876 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794337+0000", 2020-03-09T00:25:39.876 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0590000000000003e-06 2020-03-09T00:25:39.876 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.876 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.876 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:39.877 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794345+0000", 2020-03-09T00:25:39.877 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032359939999999998 2020-03-09T00:25:39.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.877 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.877 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.877 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797581+0000", 2020-03-09T00:25:39.877 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114491 2020-03-09T00:25:39.878 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.878 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.878 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.878 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797696+0000", 2020-03-09T00:25:39.878 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016180261000000001 2020-03-09T00:25:39.878 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.878 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.879 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.879 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.813876+0000", 2020-03-09T00:25:39.879 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001 2020-03-09T00:25:39.879 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.879 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.879 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.879 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.880 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.880 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:433 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~671744] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:39.880 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:39.880 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.941757545999998, 2020-03-09T00:25:39.880 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002, 2020-03-09T00:25:39.880 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.880 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.881 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.881 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.881 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.881 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 433 2020-03-09T00:25:39.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.881 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:39.882 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030767469999999999 2020-03-09T00:25:39.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.883 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.883 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.883 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804253+0000", 2020-03-09T00:25:39.883 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1990000000000006e-06 2020-03-09T00:25:39.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.883 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804262+0000", 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9800999999999999e-05 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804281+0000", 2020-03-09T00:25:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0343000000000001e-05 2020-03-09T00:25:39.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.885 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.885 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.885 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804292+0000", 2020-03-09T00:25:39.885 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000220564 2020-03-09T00:25:39.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.885 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.886 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.886 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804512+0000", 2020-03-09T00:25:39.886 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012449100000000001 2020-03-09T00:25:39.886 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.886 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.886 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.886 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804637+0000", 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00145191 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806089+0000", 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020643070999999999 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.887 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826732+0000", 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081151190000000005 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834847+0000", 2020-03-09T00:25:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1645999999999996e-05 2020-03-09T00:25:39.889 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.889 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.889 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.889 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834939+0000", 2020-03-09T00:25:39.889 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002 2020-03-09T00:25:39.889 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.889 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.890 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.890 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.890 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.890 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=66, reqid=client.4751.0:397, at_version=55'17, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:39.890 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:39.890 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.938570040999998, 2020-03-09T00:25:39.890 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001, 2020-03-09T00:25:39.891 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.891 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.891 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:39.891 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:39.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104311 2020-03-09T00:25:39.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.893 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804469+0000", 2020-03-09T00:25:39.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5208999999999999e-05 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804494+0000", 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5976000000000001e-05 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804510+0000", 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0229999999999998e-06 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804516+0000", 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057666139999999998 2020-03-09T00:25:39.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.896 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810282+0000", 2020-03-09T00:25:39.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3182e-05 2020-03-09T00:25:39.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810296+0000", 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016691906999999999 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826987+0000", 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8390000000000003e-05 2020-03-09T00:25:39.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.898 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827036+0000", 2020-03-09T00:25:39.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001 2020-03-09T00:25:39.898 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.898 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.898 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:434 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~770048] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.937828271999997, 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003, 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.899 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.900 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.900 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.900 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.900 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 434 2020-03-09T00:25:39.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.900 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:39.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011664830000000001 2020-03-09T00:25:39.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806272+0000", 2020-03-09T00:25:39.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9829999999999999e-06 2020-03-09T00:25:39.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806279+0000", 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1960000000000001e-05 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806291+0000", 2020-03-09T00:25:39.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4999999999999999e-06 2020-03-09T00:25:39.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806297+0000", 2020-03-09T00:25:39.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075845660000000001 2020-03-09T00:25:39.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.813882+0000", 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013610299999999999 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.814018+0000", 2020-03-09T00:25:39.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047231749999999996 2020-03-09T00:25:39.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.818741+0000", 2020-03-09T00:25:39.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01050245 2020-03-09T00:25:39.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829243+0000", 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0085708439999999993 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837814+0000", 2020-03-09T00:25:39.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4356000000000003e-05 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837898+0000", 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.908 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.909 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:435 3.es0 3:73e16f4f:::smithi10428818-13:head [setxattr _header (57) in=7b] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:39.909 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:39.909 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.936582498, 2020-03-09T00:25:39.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003, 2020-03-09T00:25:39.909 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 435 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4781000000000001e-05 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806427+0000", 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9479999999999996e-06 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.913 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.913 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806433+0000", 2020-03-09T00:25:39.913 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1838000000000001e-05 2020-03-09T00:25:39.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.913 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.913 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806444+0000", 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7199999999999997e-06 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806449+0000", 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020656463999999999 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827106+0000", 2020-03-09T00:25:39.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013190700000000001 2020-03-09T00:25:39.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827237+0000", 2020-03-09T00:25:39.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00078356200000000002 2020-03-09T00:25:39.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828021+0000", 2020-03-09T00:25:39.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001223678 2020-03-09T00:25:39.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829245+0000", 2020-03-09T00:25:39.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087850050000000002 2020-03-09T00:25:39.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838030+0000", 2020-03-09T00:25:39.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7771999999999998e-05 2020-03-09T00:25:39.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838127+0000", 2020-03-09T00:25:39.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:436 3.es0 3:73e16f4f:::smithi10428818-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:39.919 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.936446003999997, 2020-03-09T00:25:39.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998, 2020-03-09T00:25:39.920 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.920 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:39.920 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.920 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.920 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.920 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 436 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3732000000000001e-05 2020-03-09T00:25:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806532+0000", 2020-03-09T00:25:39.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5879999999999998e-06 2020-03-09T00:25:39.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.923 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806541+0000", 2020-03-09T00:25:39.923 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2738000000000001e-05 2020-03-09T00:25:39.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.924 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806553+0000", 2020-03-09T00:25:39.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.605e-06 2020-03-09T00:25:39.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.924 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806558+0000", 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021728470999999999 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828286+0000", 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113791 2020-03-09T00:25:39.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:39.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828400+0000", 2020-03-09T00:25:39.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010563283 2020-03-09T00:25:39.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.926 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838963+0000", 2020-03-09T00:25:39.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121599 2020-03-09T00:25:39.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.927 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.839085+0000", 2020-03-09T00:25:39.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028128480000000002 2020-03-09T00:25:39.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.841898+0000", 2020-03-09T00:25:39.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998 2020-03-09T00:25:39.928 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.928 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=67, reqid=client.4751.0:398, at_version=55'18, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.932555229999998, 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999, 2020-03-09T00:25:39.929 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.930 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:39.930 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:39.930 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000127076 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.931 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810506+0000", 2020-03-09T00:25:39.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3247999999999998e-05 2020-03-09T00:25:39.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.932 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810529+0000", 2020-03-09T00:25:39.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4992e-05 2020-03-09T00:25:39.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810544+0000", 2020-03-09T00:25:39.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9669999999999998e-06 2020-03-09T00:25:39.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810550+0000", 2020-03-09T00:25:39.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098081420000000006 2020-03-09T00:25:39.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820358+0000", 2020-03-09T00:25:39.935 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5410999999999999e-05 2020-03-09T00:25:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820374+0000", 2020-03-09T00:25:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066733160000000003 2020-03-09T00:25:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827047+0000", 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4610999999999998e-05 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827082+0000", 2020-03-09T00:25:39.937 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999 2020-03-09T00:25:39.938 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.938 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.938 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.938 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.938 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.938 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:522 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:39.938 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 26.972184583000001, 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018154887000000001, 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.939 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 522 2020-03-09T00:25:39.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.940 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.940 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.940 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:39.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7240000000000002e-05 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770797+0000", 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1880000000000002e-06 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770805+0000", 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5597e-05 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770821+0000", 2020-03-09T00:25:39.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0109999999999997e-06 2020-03-09T00:25:39.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.943 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770827+0000", 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00024047700000000001 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771067+0000", 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0305e-05 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:39.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771078+0000", 2020-03-09T00:25:39.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003975007 2020-03-09T00:25:39.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.775053+0000", 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000152447 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.775205+0000", 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010782299999999999 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.776283+0000", 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071855169999999998 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783469+0000", 2020-03-09T00:25:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0053400169999999999 2020-03-09T00:25:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.788809+0000", 2020-03-09T00:25:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5851000000000001e-05 2020-03-09T00:25:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.788905+0000", 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018154887000000001 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:523 3.es0 3:73e16f4f:::smithi10428818-13:head [rollback 1a] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 26.972062807, 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018315774999999999, 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.950 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.951 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 523 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.953 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:39.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3540000000000002e-05 2020-03-09T00:25:39.953 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.953 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.953 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770895+0000", 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7960000000000002e-06 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770899+0000", 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.922e-06 2020-03-09T00:25:39.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770906+0000", 2020-03-09T00:25:39.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9869999999999999e-06 2020-03-09T00:25:39.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770909+0000", 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000235536 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771144+0000", 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4800000000000001e-06 2020-03-09T00:25:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:39.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771153+0000", 2020-03-09T00:25:39.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0091831280000000005 2020-03-09T00:25:39.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.780336+0000", 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013816699999999999 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.780474+0000", 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00086282600000000002 2020-03-09T00:25:39.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.781337+0000", 2020-03-09T00:25:39.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002215828 2020-03-09T00:25:39.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783553+0000", 2020-03-09T00:25:39.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0055424929999999999 2020-03-09T00:25:39.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.789095+0000", 2020-03-09T00:25:39.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2071999999999995e-05 2020-03-09T00:25:39.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.789187+0000", 2020-03-09T00:25:39.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018315774999999999 2020-03-09T00:25:39.963 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.963 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.963 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.964 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:524 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:39.964 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:39.964 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 26.971995481, 2020-03-09T00:25:39.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020037222, 2020-03-09T00:25:39.964 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:39.964 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:39.965 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:39.965 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:39.965 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:39.965 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 524 2020-03-09T00:25:39.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.965 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:39.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:39.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2867999999999997e-05 2020-03-09T00:25:39.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:39.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770982+0000", 2020-03-09T00:25:39.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8850000000000004e-06 2020-03-09T00:25:39.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770988+0000", 2020-03-09T00:25:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1586000000000001e-05 2020-03-09T00:25:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:39.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770999+0000", 2020-03-09T00:25:39.969 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8590000000000004e-06 2020-03-09T00:25:39.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.969 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:39.969 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771004+0000", 2020-03-09T00:25:39.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000171402 2020-03-09T00:25:39.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771175+0000", 2020-03-09T00:25:39.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2629999999999997e-06 2020-03-09T00:25:39.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:39.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771183+0000", 2020-03-09T00:25:39.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011237143999999999 2020-03-09T00:25:39.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:39.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.782420+0000", 2020-03-09T00:25:39.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121636 2020-03-09T00:25:39.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:39.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.782541+0000", 2020-03-09T00:25:39.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00072058300000000001 2020-03-09T00:25:39.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:39.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783262+0000", 2020-03-09T00:25:39.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044685599999999998 2020-03-09T00:25:39.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:39.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783709+0000", 2020-03-09T00:25:39.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071821319999999999 2020-03-09T00:25:39.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:39.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.790891+0000", 2020-03-09T00:25:39.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5007999999999997e-05 2020-03-09T00:25:39.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:39.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:39.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:39.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.790976+0000", 2020-03-09T00:25:39.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020037222 2020-03-09T00:25:39.976 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.977 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.977 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.977 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:39.977 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:39.977 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:39.980 INFO:tasks.rados.rados.0.smithi104.stdout:343: done (0 left) 2020-03-09T00:25:39.980 INFO:tasks.rados.rados.0.smithi104.stdout:344: copy_from oid 3 from oid 37 current snap is 27 2020-03-09T00:25:39.980 INFO:tasks.rados.rados.0.smithi104.stdout:345: snap_remove snap 26 2020-03-09T00:25:39.980 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is set 2020-03-09T00:25:39.981 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:39.977+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:39.981 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:25:40.080 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:40.078+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:40.108 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:40.108 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:40.108 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:40.108 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:40.117 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:25:40.180 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:40.179+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:40.233 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:40.234 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:40.234 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:40.234 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:40.234 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:40.242 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:25:40.281 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:40.280+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:40.382 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:40.380+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:40.398 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:40.398 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:299 3.fs0 3:f75fa7c5:::smithi10428818-15:head [copy-from ver 4] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.577207281, 2020-03-09T00:25:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020952168, 2020-03-09T00:25:40.399 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 299 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818115+0000", 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012868800000000001 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818244+0000", 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.5429999999999994e-06 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.402 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818252+0000", 2020-03-09T00:25:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1410999999999999e-05 2020-03-09T00:25:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.403 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818314+0000", 2020-03-09T00:25:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9577000000000002e-05 2020-03-09T00:25:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818343+0000", 2020-03-09T00:25:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8003e-05 2020-03-09T00:25:40.404 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818401+0000", 2020-03-09T00:25:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014568900000000001 2020-03-09T00:25:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.405 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818547+0000", 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0078751659999999994 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.826422+0000", 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023378689999999998 2020-03-09T00:25:40.406 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828760+0000", 2020-03-09T00:25:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010241317999999999 2020-03-09T00:25:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.407 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839001+0000", 2020-03-09T00:25:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5903999999999998e-05 2020-03-09T00:25:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.408 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839067+0000", 2020-03-09T00:25:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020952168 2020-03-09T00:25:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.409 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:300 3.fs0 3:f75fa7c5:::smithi10428818-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.576888562999997, 2020-03-09T00:25:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021129146000000001, 2020-03-09T00:25:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.410 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 300 2020-03-09T00:25:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.411 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.412 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.413 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818434+0000", 2020-03-09T00:25:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2379000000000005e-05 2020-03-09T00:25:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.414 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818516+0000", 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8039999999999998e-06 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818522+0000", 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.208e-05 2020-03-09T00:25:40.415 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818534+0000", 2020-03-09T00:25:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5580000000000002e-06 2020-03-09T00:25:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.416 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818539+0000", 2020-03-09T00:25:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00060248300000000001 2020-03-09T00:25:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.417 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.819141+0000", 2020-03-09T00:25:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1888e-05 2020-03-09T00:25:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.418 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.819213+0000", 2020-03-09T00:25:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091353879999999995 2020-03-09T00:25:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828348+0000", 2020-03-09T00:25:40.419 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012281999999999999 2020-03-09T00:25:40.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.420 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.420 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.828471+0000", 2020-03-09T00:25:40.420 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010931410000000001 2020-03-09T00:25:40.420 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.420 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839403+0000", 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3260000000000001e-05 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839466+0000", 2020-03-09T00:25:40.421 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7076000000000004e-05 2020-03-09T00:25:40.422 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.422 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.422 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.422 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.839563+0000", 2020-03-09T00:25:40.422 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021129146000000001 2020-03-09T00:25:40.422 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.422 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:304 3.cs0 3:31ded997:::smithi10428818-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.565357081000002, 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018218758000000002, 2020-03-09T00:25:40.423 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.424 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.424 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.424 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 304 2020-03-09T00:25:40.424 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.424 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.425 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.829965+0000", 2020-03-09T00:25:40.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7318999999999999e-05 2020-03-09T00:25:40.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.426 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.426 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.426 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830013+0000", 2020-03-09T00:25:40.426 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1289999999999998e-06 2020-03-09T00:25:40.426 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.427 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830020+0000", 2020-03-09T00:25:40.427 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2313000000000001e-05 2020-03-09T00:25:40.427 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.427 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.427 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830032+0000", 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7829999999999998e-06 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.830037+0000", 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.007629449 2020-03-09T00:25:40.428 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.429 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837666+0000", 2020-03-09T00:25:40.429 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1544000000000003e-05 2020-03-09T00:25:40.429 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.429 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.429 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.837758+0000", 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0089964079999999991 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846754+0000", 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113248 2020-03-09T00:25:40.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.431 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.846867+0000", 2020-03-09T00:25:40.431 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001316565 2020-03-09T00:25:40.431 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.431 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.431 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.848184+0000", 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018218758000000002 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.432 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:344 3.fs0 3:f2571ca0:::smithi10428818-48:head [copy-from ver 35] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:40.433 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:40.433 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.583511170999998, 2020-03-09T00:25:40.433 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066389322000000001, 2020-03-09T00:25:40.433 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.433 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.433 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.433 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 344 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:40.434 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811811+0000", 2020-03-09T00:25:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100208 2020-03-09T00:25:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811911+0000", 2020-03-09T00:25:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1372999999999999e-05 2020-03-09T00:25:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.436 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811923+0000", 2020-03-09T00:25:40.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5517999999999999e-05 2020-03-09T00:25:40.437 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.437 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.437 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.437 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811938+0000", 2020-03-09T00:25:40.437 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7190000000000004e-06 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.811945+0000", 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1340000000000006e-05 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.438 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812006+0000", 2020-03-09T00:25:40.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014919999999999999 2020-03-09T00:25:40.439 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.439 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.439 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.439 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812156+0000", 2020-03-09T00:25:40.439 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.038728380999999999 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.850884+0000", 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013700324 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864584+0000", 2020-03-09T00:25:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013548961 2020-03-09T00:25:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878133+0000", 2020-03-09T00:25:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7298000000000005e-05 2020-03-09T00:25:40.442 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.443 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.443 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.444 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878200+0000", 2020-03-09T00:25:40.444 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066389322000000001 2020-03-09T00:25:40.444 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.444 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.444 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.444 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.444 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.445 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:345 3.fs0 3:f2571ca0:::smithi10428818-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:40.445 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:40.445 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 37.583322588999998, 2020-03-09T00:25:40.445 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066989273000000002, 2020-03-09T00:25:40.445 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.445 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.445 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 345 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.446 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:40.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.447 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.447 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.447 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812000+0000", 2020-03-09T00:25:40.447 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5945000000000001e-05 2020-03-09T00:25:40.447 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.448 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812046+0000", 2020-03-09T00:25:40.448 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.545e-06 2020-03-09T00:25:40.448 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.448 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.448 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812052+0000", 2020-03-09T00:25:40.449 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3108999999999999e-05 2020-03-09T00:25:40.449 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.449 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.449 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.449 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812065+0000", 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.566e-06 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812070+0000", 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00034552600000000003 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.450 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812415+0000", 2020-03-09T00:25:40.451 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8084999999999997e-05 2020-03-09T00:25:40.451 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.451 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.451 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:40.451 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.812474+0000", 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.051585182 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864059+0000", 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8030999999999999e-05 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.452 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.864137+0000", 2020-03-09T00:25:40.453 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014720627 2020-03-09T00:25:40.453 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.453 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.453 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.453 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878857+0000", 2020-03-09T00:25:40.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9045999999999997e-05 2020-03-09T00:25:40.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.454 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.454 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.454 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878916+0000", 2020-03-09T00:25:40.454 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2611e-05 2020-03-09T00:25:40.454 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.455 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.455 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.455 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.878989+0000", 2020-03-09T00:25:40.455 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.066989273000000002 2020-03-09T00:25:40.455 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.455 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.456 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.456 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.456 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.456 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:380 3.fs0 3:fa16d489:::smithi10428818-27:head [append 0~557056] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:40.456 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:40.457 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.583482617000001, 2020-03-09T00:25:40.457 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019518607, 2020-03-09T00:25:40.457 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.457 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.457 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.457 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.457 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.458 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 380 2020-03-09T00:25:40.458 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.458 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.458 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.458 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.458 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:40.458 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.459 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.459 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.811840+0000", 2020-03-09T00:25:40.459 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036718500000000002 2020-03-09T00:25:40.459 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.459 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.460 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812207+0000", 2020-03-09T00:25:40.460 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.3760000000000008e-06 2020-03-09T00:25:40.460 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.460 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.460 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812215+0000", 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5866999999999999e-05 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812231+0000", 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3609999999999997e-06 2020-03-09T00:25:40.461 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.462 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812238+0000", 2020-03-09T00:25:40.462 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6171000000000001e-05 2020-03-09T00:25:40.462 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.462 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.462 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812304+0000", 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.8213000000000003e-05 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.812402+0000", 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012745790000000001 2020-03-09T00:25:40.463 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.464 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.813676+0000", 2020-03-09T00:25:40.464 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095836270000000008 2020-03-09T00:25:40.464 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.464 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.464 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.823260+0000", 2020-03-09T00:25:40.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0080215159999999994 2020-03-09T00:25:40.465 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.465 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.465 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.465 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831282+0000", 2020-03-09T00:25:40.465 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6711999999999993e-05 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.831358+0000", 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.019518607 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.466 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.467 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.467 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.467 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:384 3.fs0 3:fa16d489:::smithi10428818-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:40.467 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:40.467 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.577220574999998, 2020-03-09T00:25:40.467 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018476747000000002, 2020-03-09T00:25:40.467 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 384 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.468 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.469 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.469 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:40.469 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.469 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.469 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818102+0000", 2020-03-09T00:25:40.470 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1008999999999999e-05 2020-03-09T00:25:40.470 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.470 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.470 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.470 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818153+0000", 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2199999999999997e-06 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818159+0000", 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3512999999999999e-05 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.471 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818173+0000", 2020-03-09T00:25:40.472 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5650000000000003e-06 2020-03-09T00:25:40.472 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.472 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.472 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.472 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818177+0000", 2020-03-09T00:25:40.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01043207 2020-03-09T00:25:40.473 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.473 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.473 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.473 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.828609+0000", 2020-03-09T00:25:40.473 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000216252 2020-03-09T00:25:40.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.474 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.474 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.828825+0000", 2020-03-09T00:25:40.474 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005000539 2020-03-09T00:25:40.474 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.474 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833826+0000", 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9849000000000001e-05 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.833906+0000", 2020-03-09T00:25:40.475 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026727299999999999 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.836579+0000", 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018476747000000002 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.476 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.477 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.477 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.477 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:393 3.fs0 3:f1702e73:::smithi10428818-39:10 [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:40.477 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:40.477 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.576763751999998, 2020-03-09T00:25:40.477 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01890696, 2020-03-09T00:25:40.477 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.478 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.478 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.478 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 393 2020-03-09T00:25:40.478 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.479 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.479 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.479 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.479 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:40.479 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.480 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.481 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.481 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818559+0000", 2020-03-09T00:25:40.481 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1644000000000003e-05 2020-03-09T00:25:40.481 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.481 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.482 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818600+0000", 2020-03-09T00:25:40.482 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8680000000000001e-06 2020-03-09T00:25:40.482 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.482 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.482 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818604+0000", 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1230000000000002e-06 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818611+0000", 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2399999999999999e-06 2020-03-09T00:25:40.483 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.484 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.818614+0000", 2020-03-09T00:25:40.484 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015764628999999999 2020-03-09T00:25:40.484 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.484 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.484 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.834379+0000", 2020-03-09T00:25:40.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016986400000000001 2020-03-09T00:25:40.485 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.485 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.485 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.485 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.834549+0000", 2020-03-09T00:25:40.485 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002916592 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.837466+0000", 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01890696 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.486 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.487 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.487 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:544 3.fs0 3:f75fa7c5:::smithi10428818-15:head [append 0~491520] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:40.487 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:13.787862+0000", 2020-03-09T00:25:40.487 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.607459976000001, 2020-03-09T00:25:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022206677000000001, 2020-03-09T00:25:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.488 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 544 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787862+0000", 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.490 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.787862+0000", 2020-03-09T00:25:40.490 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00027661299999999999 2020-03-09T00:25:40.490 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.490 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.490 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788139+0000", 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9300000000000003e-06 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788147+0000", 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5483999999999999e-05 2020-03-09T00:25:40.491 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.492 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788162+0000", 2020-03-09T00:25:40.492 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4910000000000002e-06 2020-03-09T00:25:40.492 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.492 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.492 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788169+0000", 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3137999999999998e-05 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788222+0000", 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110845 2020-03-09T00:25:40.493 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.494 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.788333+0000", 2020-03-09T00:25:40.494 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012354709999999999 2020-03-09T00:25:40.494 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.494 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.494 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.789568+0000", 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003034138 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.792602+0000", 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017400841 2020-03-09T00:25:40.495 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.810003+0000", 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5726000000000001e-05 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.496 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:13.810069+0000", 2020-03-09T00:25:40.497 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.022206677000000001 2020-03-09T00:25:40.497 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.497 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.497 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.497 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.497 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.497 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:551 3.cs0 3:3776f87d:::smithi10428818-35:head [copy-from ver 24] snapc 1b=[1a,13,c] ondisk+write+known_if_redirected e65)", 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:14.774998+0000", 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.620324441000001, 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.048260305000000003, 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.498 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.499 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 551 2020-03-09T00:25:40.499 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.499 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.499 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.499 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.499 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.774998+0000", 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.774998+0000", 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3563999999999998e-05 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.500 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775041+0000", 2020-03-09T00:25:40.501 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2859999999999996e-06 2020-03-09T00:25:40.501 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.501 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.501 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.501 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775048+0000", 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1956e-05 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775060+0000", 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8690000000000004e-06 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.502 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775068+0000", 2020-03-09T00:25:40.503 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0949000000000002e-05 2020-03-09T00:25:40.503 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.503 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.503 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.503 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775129+0000", 2020-03-09T00:25:40.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3609e-05 2020-03-09T00:25:40.504 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.504 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.504 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:40.504 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775142+0000", 2020-03-09T00:25:40.504 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023812970000000001 2020-03-09T00:25:40.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.505 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.505 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.505 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777523+0000", 2020-03-09T00:25:40.505 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5765000000000002e-05 2020-03-09T00:25:40.505 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.506 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.506 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777619+0000", 2020-03-09T00:25:40.507 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024822420000000001 2020-03-09T00:25:40.507 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.507 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.507 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.507 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.802442+0000", 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012621762999999999 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.815063+0000", 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081185110000000001 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.508 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.823182+0000", 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6316000000000006e-05 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.823258+0000", 2020-03-09T00:25:40.509 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.048260305000000003 2020-03-09T00:25:40.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.510 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.510 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.510 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.510 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.510 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:552 3.cs0 3:3776f87d:::smithi10428818-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e65)", 2020-03-09T00:25:40.510 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:14.775120+0000", 2020-03-09T00:25:40.511 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.620202101, 2020-03-09T00:25:40.511 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.049740894000000001, 2020-03-09T00:25:40.511 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.511 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.511 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.511 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.511 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 552 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775120+0000", 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.512 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.513 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775120+0000", 2020-03-09T00:25:40.513 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4431000000000002e-05 2020-03-09T00:25:40.513 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.513 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.513 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775165+0000", 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7569999999999998e-06 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775171+0000", 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3426e-05 2020-03-09T00:25:40.514 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.515 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775185+0000", 2020-03-09T00:25:40.515 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2839999999999997e-06 2020-03-09T00:25:40.515 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.515 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.515 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.516 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775190+0000", 2020-03-09T00:25:40.516 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9645000000000002e-05 2020-03-09T00:25:40.516 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.517 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775250+0000", 2020-03-09T00:25:40.517 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0207000000000001e-05 2020-03-09T00:25:40.517 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.517 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.517 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.775260+0000", 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026141129999999999 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777874+0000", 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3913999999999999e-05 2020-03-09T00:25:40.518 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:40.519 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.777928+0000", 2020-03-09T00:25:40.519 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.036313404000000001 2020-03-09T00:25:40.519 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.519 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.519 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.814241+0000", 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6928999999999994e-05 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.814328+0000", 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010364329 2020-03-09T00:25:40.520 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.521 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.824693+0000", 2020-03-09T00:25:40.521 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4103000000000006e-05 2020-03-09T00:25:40.521 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.521 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.521 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.824757+0000", 2020-03-09T00:25:40.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010435200000000001 2020-03-09T00:25:40.522 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.522 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.522 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.522 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:14.824861+0000", 2020-03-09T00:25:40.522 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.049740894000000001 2020-03-09T00:25:40.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.523 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.523 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.523 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.523 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.523 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:572 3.cs0 3:3776f87d:::smithi10428818-35:head [copy-from ver 6] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:40.523 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.943798+0000", 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.451524639999999, 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021300082000000001, 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 572 2020-03-09T00:25:40.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.525 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.525 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.525 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.525 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943798+0000", 2020-03-09T00:25:40.525 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.525 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943798+0000", 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1941000000000003e-05 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.526 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943850+0000", 2020-03-09T00:25:40.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.7620000000000001e-06 2020-03-09T00:25:40.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.527 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.527 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.527 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943855+0000", 2020-03-09T00:25:40.527 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4925e-05 2020-03-09T00:25:40.527 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943870+0000", 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9259999999999999e-06 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.528 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943875+0000", 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9487000000000003e-05 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943915+0000", 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012597699999999999 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.529 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.944041+0000", 2020-03-09T00:25:40.530 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003912436 2020-03-09T00:25:40.530 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.530 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.530 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.530 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.947953+0000", 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033233059999999998 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.951276+0000", 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013528966999999999 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.531 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.964805+0000", 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000292355 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965098+0000", 2020-03-09T00:25:40.532 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021300082000000001 2020-03-09T00:25:40.533 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.533 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.533 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.533 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.533 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.533 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:573 3.cs0 3:3776f87d:::smithi10428818-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:40.533 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.943913+0000", 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 22.451409321, 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021618235, 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.534 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 573 2020-03-09T00:25:40.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.535 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.535 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.535 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.535 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943913+0000", 2020-03-09T00:25:40.535 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.535 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.536 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.536 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943913+0000", 2020-03-09T00:25:40.536 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6050999999999998e-05 2020-03-09T00:25:40.536 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.536 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943939+0000", 2020-03-09T00:25:40.537 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.557e-06 2020-03-09T00:25:40.537 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.537 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.537 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.537 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943943+0000", 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1754e-05 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943954+0000", 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2179999999999999e-06 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.538 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.943958+0000", 2020-03-09T00:25:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021570000000000001 2020-03-09T00:25:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.539 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.944173+0000", 2020-03-09T00:25:40.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1937000000000002e-05 2020-03-09T00:25:40.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.540 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.540 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:40.540 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.944215+0000", 2020-03-09T00:25:40.540 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069155550000000003 2020-03-09T00:25:40.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.951131+0000", 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013795300000000001 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.541 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.951269+0000", 2020-03-09T00:25:40.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013891163999999999 2020-03-09T00:25:40.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.542 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.542 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.542 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965160+0000", 2020-03-09T00:25:40.542 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011575500000000001 2020-03-09T00:25:40.542 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965276+0000", 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000255591 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.543 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.965531+0000", 2020-03-09T00:25:40.544 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.021618235 2020-03-09T00:25:40.544 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.544 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.544 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.544 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.544 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.544 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:634 3.cs0 3:32d16cfa:::smithi10428818-24:head [copy-from ver 4] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:40.545 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.778929+0000", 2020-03-09T00:25:40.545 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.616393451, 2020-03-09T00:25:40.545 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027938032000000002, 2020-03-09T00:25:40.545 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.545 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.545 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.545 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.546 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.546 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 634 2020-03-09T00:25:40.546 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.546 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.546 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.546 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778929+0000", 2020-03-09T00:25:40.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.547 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.547 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.547 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778929+0000", 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0008e-05 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778969+0000", 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9050000000000002e-06 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778975+0000", 2020-03-09T00:25:40.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0200000000000001e-05 2020-03-09T00:25:40.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778985+0000", 2020-03-09T00:25:40.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1460000000000001e-06 2020-03-09T00:25:40.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.778991+0000", 2020-03-09T00:25:40.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0701000000000001e-05 2020-03-09T00:25:40.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779042+0000", 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6040000000000004e-06 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:40.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779049+0000", 2020-03-09T00:25:40.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0043638970000000003 2020-03-09T00:25:40.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783413+0000", 2020-03-09T00:25:40.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013153199999999999 2020-03-09T00:25:40.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783545+0000", 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016497833 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.800043+0000", 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055694259999999997 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805612+0000", 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010821209999999999 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806694+0000", 2020-03-09T00:25:40.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000172659 2020-03-09T00:25:40.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.555 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.555 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806867+0000", 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027938032000000002 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:635 3.cs0 3:32d16cfa:::smithi10428818-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e75)", 2020-03-09T00:25:40.556 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.779023+0000", 2020-03-09T00:25:40.557 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.616299569000001, 2020-03-09T00:25:40.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.028096716000000001, 2020-03-09T00:25:40.557 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.557 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.557 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.557 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.557 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 635 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779023+0000", 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779023+0000", 2020-03-09T00:25:40.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4238000000000001e-05 2020-03-09T00:25:40.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779047+0000", 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4230000000000001e-06 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779050+0000", 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2509999999999997e-06 2020-03-09T00:25:40.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779058+0000", 2020-03-09T00:25:40.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.785e-06 2020-03-09T00:25:40.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779060+0000", 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0540999999999997e-05 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779101+0000", 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.5169999999999999e-06 2020-03-09T00:25:40.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:40.563 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.779107+0000", 2020-03-09T00:25:40.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046296230000000002 2020-03-09T00:25:40.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.563 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783736+0000", 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1681000000000002e-05 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.783788+0000", 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020864237000000001 2020-03-09T00:25:40.564 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804652+0000", 2020-03-09T00:25:40.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110419 2020-03-09T00:25:40.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804762+0000", 2020-03-09T00:25:40.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021448970000000002 2020-03-09T00:25:40.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806907+0000", 2020-03-09T00:25:40.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3344999999999997e-05 2020-03-09T00:25:40.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806971+0000", 2020-03-09T00:25:40.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014875899999999999 2020-03-09T00:25:40.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.807119+0000", 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.028096716000000001 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.569 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:639 3.cs0 3:31ded997:::smithi10428818-32:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:40.569 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785921+0000", 2020-03-09T00:25:40.569 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.609400945000001, 2020-03-09T00:25:40.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020119509000000001, 2020-03-09T00:25:40.569 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.569 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.569 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.570 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.570 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.570 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 639 2020-03-09T00:25:40.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.571 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.572 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785921+0000", 2020-03-09T00:25:40.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.572 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785921+0000", 2020-03-09T00:25:40.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4159000000000002e-05 2020-03-09T00:25:40.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785966+0000", 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.844e-06 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785972+0000", 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2754e-05 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785985+0000", 2020-03-09T00:25:40.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1830000000000002e-06 2020-03-09T00:25:40.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785990+0000", 2020-03-09T00:25:40.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00053932700000000004 2020-03-09T00:25:40.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786530+0000", 2020-03-09T00:25:40.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018090599999999999 2020-03-09T00:25:40.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786711+0000", 2020-03-09T00:25:40.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0081753880000000004 2020-03-09T00:25:40.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.794886+0000", 2020-03-09T00:25:40.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010617161 2020-03-09T00:25:40.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805503+0000", 2020-03-09T00:25:40.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039746399999999999 2020-03-09T00:25:40.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805901+0000", 2020-03-09T00:25:40.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000140323 2020-03-09T00:25:40.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.580 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.580 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806041+0000", 2020-03-09T00:25:40.580 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020119509000000001 2020-03-09T00:25:40.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.580 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.581 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:640 3.cs0 3:31ded997:::smithi10428818-32:head [rollback 23] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:40.581 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.786041+0000", 2020-03-09T00:25:40.581 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.609281692, 2020-03-09T00:25:40.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020266050000000001, 2020-03-09T00:25:40.581 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.581 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.582 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.582 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.582 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.582 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 640 2020-03-09T00:25:40.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.582 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786041+0000", 2020-03-09T00:25:40.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786041+0000", 2020-03-09T00:25:40.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6978999999999999e-05 2020-03-09T00:25:40.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786088+0000", 2020-03-09T00:25:40.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4779999999999996e-06 2020-03-09T00:25:40.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786094+0000", 2020-03-09T00:25:40.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1922e-05 2020-03-09T00:25:40.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786106+0000", 2020-03-09T00:25:40.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6349999999999997e-06 2020-03-09T00:25:40.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786111+0000", 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0094916340000000005 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.795602+0000", 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013348799999999999 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.795736+0000", 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082019099999999995 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796556+0000", 2020-03-09T00:25:40.589 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0090484759999999997 2020-03-09T00:25:40.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.589 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.589 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805604+0000", 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00054049499999999999 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806145+0000", 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016175200000000001 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.806307+0000", 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020266050000000001 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:641 3.cs0 3:31ded997:::smithi10428818-32:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.786157+0000", 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 15.609165823, 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024528417, 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.592 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.593 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 641 2020-03-09T00:25:40.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.593 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786157+0000", 2020-03-09T00:25:40.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786157+0000", 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2144000000000002e-05 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786199+0000", 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1720000000000004e-06 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786205+0000", 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2237e-05 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786217+0000", 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5460000000000002e-06 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.786222+0000", 2020-03-09T00:25:40.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018580409999999999 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804802+0000", 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3192e-05 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.597 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.804875+0000", 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046342300000000003 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805339+0000", 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00039975600000000003 2020-03-09T00:25:40.598 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.805738+0000", 2020-03-09T00:25:40.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048028309999999996 2020-03-09T00:25:40.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.810541+0000", 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014370600000000001 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.810685+0000", 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.024528417 2020-03-09T00:25:40.600 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.601 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.601 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.601 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:758 3.1s1 3:85442c74:::smithi10428818-19:head [copy-from ver 24] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:40.601 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:34.872442+0000", 2020-03-09T00:25:40.601 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.5228801909999996, 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.033435698, 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 758 2020-03-09T00:25:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872442+0000", 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872442+0000", 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6240000000000001e-05 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872528+0000", 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7886000000000001e-05 2020-03-09T00:25:40.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872546+0000", 2020-03-09T00:25:40.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9837e-05 2020-03-09T00:25:40.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872566+0000", 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1200000000000008e-06 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872575+0000", 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.2102999999999998e-05 2020-03-09T00:25:40.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872637+0000", 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000210196 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872848+0000", 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012013687 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.884861+0000", 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012060997 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.896922+0000", 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0086458330000000003 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.905568+0000", 2020-03-09T00:25:40.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030979899999999998 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.905878+0000", 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.033435698 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.610 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.611 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:759 3.1s1 3:85442c74:::smithi10428818-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e85)", 2020-03-09T00:25:40.611 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:34.872630+0000", 2020-03-09T00:25:40.611 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.5226921730000003, 2020-03-09T00:25:40.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.034372448999999999, 2020-03-09T00:25:40.611 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 759 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:40.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:40.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872630+0000", 2020-03-09T00:25:40.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:40.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:40.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872630+0000", 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4660999999999998e-05 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872675+0000", 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4169999999999997e-06 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872681+0000", 2020-03-09T00:25:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2785999999999999e-05 2020-03-09T00:25:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872694+0000", 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2460000000000003e-06 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.872699+0000", 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00036016599999999998 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.873059+0000", 2020-03-09T00:25:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9513999999999999e-05 2020-03-09T00:25:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.873129+0000", 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023831893 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.896961+0000", 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013303999999999999 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.897094+0000", 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0096604099999999995 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.906754+0000", 2020-03-09T00:25:40.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0538999999999996e-05 2020-03-09T00:25:40.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:40.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.906835+0000", 2020-03-09T00:25:40.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000167777 2020-03-09T00:25:40.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:40.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:34.907003+0000", 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.034372448999999999 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:40.621 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:40.622 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:40.623 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:25:40.624 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:40.622+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:40.687 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:40.724 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:40.723+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:40.750 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:40.750 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:40.750 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:40.750 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:40.759 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:25:40.825 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:40.826 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:40.824+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:40.846 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:40.847 INFO:tasks.rados.rados.0.smithi104.stdout:345: done (1 left) 2020-03-09T00:25:40.848 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 17 current snap is 27 2020-03-09T00:25:40.848 INFO:tasks.rados.rados.0.smithi104.stdout:346: seq_num 137 ranges {0=753664,753664=786432,1540096=655360,2195456=114688} 2020-03-09T00:25:40.851 INFO:tasks.rados.rados.0.smithi104.stdout:346: writing smithi10428818-17 from 0 to 753664 tid 1 2020-03-09T00:25:40.856 INFO:tasks.rados.rados.0.smithi104.stdout:346: writing smithi10428818-17 from 753664 to 1540096 tid 2 2020-03-09T00:25:40.860 INFO:tasks.rados.rados.0.smithi104.stdout:346: writing smithi10428818-17 from 1540096 to 2195456 tid 3 2020-03-09T00:25:40.862 INFO:tasks.rados.rados.0.smithi104.stdout:346: writing smithi10428818-17 from 2195456 to 2310144 tid 4 2020-03-09T00:25:40.862 INFO:tasks.rados.rados.0.smithi104.stdout:347: rollback oid 16 current snap is 27 2020-03-09T00:25:40.862 INFO:tasks.rados.rados.0.smithi104.stdout:347: done (2 left) 2020-03-09T00:25:40.862 INFO:tasks.rados.rados.0.smithi104.stdout:348: snap_create 2020-03-09T00:25:40.862 INFO:tasks.rados.rados.0.smithi104.stdout:344: finishing copy_from to smithi10428818-3 2020-03-09T00:25:40.863 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 3 v 135 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-03-09T00:25:40.863 INFO:tasks.rados.rados.0.smithi104.stdout:344: finishing copy_from racing read to smithi10428818-3 2020-03-09T00:25:40.866 INFO:tasks.rados.rados.0.smithi104.stdout:346: finishing write tid 1 to smithi10428818-17 2020-03-09T00:25:40.870 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:40.871 INFO:tasks.rados.rados.0.smithi104.stdout:346: finishing write tid 2 to smithi10428818-17 2020-03-09T00:25:40.874 INFO:tasks.rados.rados.0.smithi104.stdout:346: finishing write tid 3 to smithi10428818-17 2020-03-09T00:25:40.875 INFO:tasks.rados.rados.0.smithi104.stdout:346: finishing write tid 4 to smithi10428818-17 2020-03-09T00:25:40.875 INFO:tasks.rados.rados.0.smithi104.stdout:346: finishing write tid 6 to smithi10428818-17 2020-03-09T00:25:40.877 INFO:tasks.rados.rados.0.smithi104.stdout:346: finishing write tid 7 to smithi10428818-17 2020-03-09T00:25:40.878 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 17 v 76 (ObjNum 137 snap 27 seq_num 137) dirty exists 2020-03-09T00:25:40.878 INFO:tasks.rados.rados.0.smithi104.stdout:346: left oid 17 (ObjNum 137 snap 27 seq_num 137) 2020-03-09T00:25:40.879 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:40.879 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:40.879 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:40.879 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:40.880 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:40.887 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:25:40.892 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:40.911 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:40.926 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:40.925+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:40.931 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:40.952 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:40.971 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:40.991 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:41.011 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:41.027 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:41.026+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:41.031 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:41.042 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.042 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.050 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:25:41.051 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:41.071 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:41.091 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:41.111 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:41.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.112 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.113 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:41.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:41.113 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.114 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.115 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:41.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:41.162 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:41.161+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:41.187 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.187 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.196 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:25:41.229 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:41.227+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:41.320 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.320 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.330 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:25:41.331 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:41.328+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:41.430 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:41.429+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:41.454 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.455 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.463 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:25:41.531 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:41.530+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:41.580 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.581 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.589 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:25:41.632 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:41.631+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:41.704 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.705 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.713 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:25:41.758 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:41.757+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:41.833 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.833 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.835 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:41.834+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:41.841 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:25:41.852 INFO:tasks.rados.rados.0.smithi104.stdout:344: done (2 left) 2020-03-09T00:25:41.852 INFO:tasks.rados.rados.0.smithi104.stdout:346: done (1 left) 2020-03-09T00:25:41.852 INFO:tasks.rados.rados.0.smithi104.stdout:348: done (0 left) 2020-03-09T00:25:41.852 INFO:tasks.rados.rados.0.smithi104.stdout:349: rmattr oid 24 current snap is 28 2020-03-09T00:25:41.852 INFO:tasks.rados.rados.0.smithi104.stdout:349: done (0 left) 2020-03-09T00:25:41.853 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 39 current snap is 28 2020-03-09T00:25:41.853 INFO:tasks.rados.rados.0.smithi104.stdout:350: seq_num 138 ranges {1835008=770048,2605056=540672,3145728=753664,3899392=262144} 2020-03-09T00:25:41.867 INFO:tasks.rados.rados.0.smithi104.stdout:350: writing smithi10428818-39 from 1835008 to 2605056 tid 1 2020-03-09T00:25:41.870 INFO:tasks.rados.rados.0.smithi104.stdout:350: writing smithi10428818-39 from 2605056 to 3145728 tid 2 2020-03-09T00:25:41.875 INFO:tasks.rados.rados.0.smithi104.stdout:350: writing smithi10428818-39 from 3145728 to 3899392 tid 3 2020-03-09T00:25:41.881 INFO:tasks.rados.rados.0.smithi104.stdout:350: writing smithi10428818-39 from 3899392 to 4161536 tid 4 2020-03-09T00:25:41.881 INFO:tasks.rados.rados.0.smithi104.stdout:351: setattr oid 46 current snap is 28 2020-03-09T00:25:41.881 INFO:tasks.rados.rados.0.smithi104.stdout:352: read oid 16 snap -1 2020-03-09T00:25:41.881 INFO:tasks.rados.rados.0.smithi104.stdout:352: expect (ObjNum 89 snap 14 seq_num 89) 2020-03-09T00:25:41.881 INFO:tasks.rados.rados.0.smithi104.stdout:353: delete oid 37 current snap is 28 2020-03-09T00:25:41.889 INFO:tasks.rados.rados.0.smithi104.stdout:350: finishing write tid 1 to smithi10428818-39 2020-03-09T00:25:41.902 INFO:tasks.rados.rados.0.smithi104.stdout:350: finishing write tid 2 to smithi10428818-39 2020-03-09T00:25:41.902 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 46 v 62 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-03-09T00:25:41.902 INFO:tasks.rados.rados.0.smithi104.stdout:350: finishing write tid 3 to smithi10428818-39 2020-03-09T00:25:41.902 INFO:tasks.rados.rados.0.smithi104.stdout:352: expect (ObjNum 89 snap 14 seq_num 89) 2020-03-09T00:25:41.936 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:41.935+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:41.959 INFO:tasks.rados.rados.0.smithi104.stdout:350: finishing write tid 4 to smithi10428818-39 2020-03-09T00:25:41.959 INFO:tasks.rados.rados.0.smithi104.stdout:350: finishing write tid 6 to smithi10428818-39 2020-03-09T00:25:41.959 INFO:tasks.rados.rados.0.smithi104.stdout:350: finishing write tid 7 to smithi10428818-39 2020-03-09T00:25:41.959 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 39 v 118 (ObjNum 138 snap 28 seq_num 138) dirty exists 2020-03-09T00:25:41.959 INFO:tasks.rados.rados.0.smithi104.stdout:350: left oid 39 (ObjNum 138 snap 28 seq_num 138) 2020-03-09T00:25:41.959 INFO:tasks.rados.rados.0.smithi104.stdout:350: done (3 left) 2020-03-09T00:25:41.959 INFO:tasks.rados.rados.0.smithi104.stdout:351: done (2 left) 2020-03-09T00:25:41.960 INFO:tasks.rados.rados.0.smithi104.stdout:352: done (1 left) 2020-03-09T00:25:41.960 INFO:tasks.rados.rados.0.smithi104.stdout:353: done (0 left) 2020-03-09T00:25:41.960 INFO:tasks.rados.rados.0.smithi104.stdout:354: delete oid 50 current snap is 28 2020-03-09T00:25:41.965 INFO:tasks.rados.rados.0.smithi104.stdout:354: done (0 left) 2020-03-09T00:25:41.965 INFO:tasks.rados.rados.0.smithi104.stdout:355: rollback oid 37 current snap is 28 2020-03-09T00:25:41.965 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 37 to 27 2020-03-09T00:25:41.966 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 48 current snap is 28 2020-03-09T00:25:41.966 INFO:tasks.rados.rados.0.smithi104.stdout:356: seq_num 141 ranges {3014656=425984,3440640=573440,4014080=425984,4440064=524288,4964352=425984} 2020-03-09T00:25:41.981 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set nodeep-scrub 2020-03-09T00:25:41.982 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:41.982 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:41.984 INFO:tasks.rados.rados.0.smithi104.stdout:356: writing smithi10428818-48 from 3014656 to 3440640 tid 1 2020-03-09T00:25:41.988 INFO:tasks.rados.rados.0.smithi104.stdout:356: writing smithi10428818-48 from 3440640 to 4014080 tid 2 2020-03-09T00:25:41.991 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:25:41.991 INFO:tasks.rados.rados.0.smithi104.stdout:356: writing smithi10428818-48 from 4014080 to 4440064 tid 3 2020-03-09T00:25:41.994 INFO:tasks.rados.rados.0.smithi104.stdout:356: writing smithi10428818-48 from 4440064 to 4964352 tid 4 2020-03-09T00:25:41.996 INFO:tasks.rados.rados.0.smithi104.stdout:356: writing smithi10428818-48 from 4964352 to 5390336 tid 5 2020-03-09T00:25:41.996 INFO:tasks.rados.rados.0.smithi104.stdout:357: copy_from oid 17 from oid 47 current snap is 28 2020-03-09T00:25:41.996 INFO:tasks.rados.rados.0.smithi104.stdout:355: finishing rollback tid 1 to smithi10428818-37 2020-03-09T00:25:41.997 INFO:tasks.rados.rados.0.smithi104.stdout:355: finishing rollback tid 2 to smithi10428818-37 2020-03-09T00:25:41.997 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 37 v 168 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-03-09T00:25:41.997 INFO:tasks.rados.rados.0.smithi104.stdout:356: finishing write tid 1 to smithi10428818-48 2020-03-09T00:25:41.997 INFO:tasks.rados.rados.0.smithi104.stdout:355: done (2 left) 2020-03-09T00:25:41.997 INFO:tasks.rados.rados.0.smithi104.stdout:358: snap_create 2020-03-09T00:25:42.002 INFO:tasks.rados.rados.0.smithi104.stdout:356: finishing write tid 2 to smithi10428818-48 2020-03-09T00:25:42.006 INFO:tasks.rados.rados.0.smithi104.stdout:356: finishing write tid 3 to smithi10428818-48 2020-03-09T00:25:42.006 INFO:tasks.rados.rados.0.smithi104.stdout:356: finishing write tid 4 to smithi10428818-48 2020-03-09T00:25:42.010 INFO:tasks.rados.rados.0.smithi104.stdout:356: finishing write tid 5 to smithi10428818-48 2020-03-09T00:25:42.010 INFO:tasks.rados.rados.0.smithi104.stdout:356: finishing write tid 7 to smithi10428818-48 2020-03-09T00:25:42.020 INFO:tasks.rados.rados.0.smithi104.stdout:356: finishing write tid 8 to smithi10428818-48 2020-03-09T00:25:42.020 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 48 v 125 (ObjNum 141 snap 28 seq_num 141) dirty exists 2020-03-09T00:25:42.020 INFO:tasks.rados.rados.0.smithi104.stdout:356: left oid 48 (ObjNum 141 snap 28 seq_num 141) 2020-03-09T00:25:42.026 INFO:tasks.rados.rados.0.smithi104.stdout:357: finishing copy_from to smithi10428818-17 2020-03-09T00:25:42.026 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 17 v 146 (ObjNum 132 snap 26 seq_num 132) dirty exists 2020-03-09T00:25:42.027 INFO:tasks.rados.rados.0.smithi104.stdout:357: finishing copy_from racing read to smithi10428818-17 2020-03-09T00:25:42.037 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:42.035+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:42.113 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:42.113 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:42.121 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:25:42.162 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:42.161+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:42.239 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:42.237+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:42.276 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:42.276 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:42.284 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:25:42.339 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:42.338+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:42.407 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:42.407 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:42.415 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:25:42.441 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:42.439+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:42.531 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:42.531 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:42.539 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:25:42.541 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:42.540+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:42.642 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:42.641+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:42.654 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:42.654 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:42.663 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:25:42.743 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:42.741+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:42.799 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:42.799 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:42.808 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:25:42.851 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:42.850+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is set 2020-03-09T00:25:42.857 INFO:tasks.rados.rados.0.smithi104.stdout:356: done (2 left) 2020-03-09T00:25:42.857 INFO:tasks.rados.rados.0.smithi104.stdout:357: done (1 left) 2020-03-09T00:25:42.857 INFO:tasks.rados.rados.0.smithi104.stdout:358: done (0 left) 2020-03-09T00:25:42.857 INFO:tasks.rados.rados.0.smithi104.stdout:359: setattr oid 2 current snap is 29 2020-03-09T00:25:42.859 INFO:tasks.rados.rados.0.smithi104.stdout:360: delete oid 9 current snap is 29 2020-03-09T00:25:42.861 INFO:tasks.rados.rados.0.smithi104.stdout:360: done (1 left) 2020-03-09T00:25:42.861 INFO:tasks.rados.rados.0.smithi104.stdout:361: rmattr oid 6 current snap is 29 2020-03-09T00:25:42.862 INFO:tasks.rados.rados.0.smithi104.stdout:361: done (1 left) 2020-03-09T00:25:42.862 INFO:tasks.rados.rados.0.smithi104.stdout:362: snap_create 2020-03-09T00:25:42.869 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 2 v 170 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-03-09T00:25:42.936 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:42.936 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:42.944 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:25:42.945 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:42.943+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:43.046 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:43.044+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:43.108 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:43.108 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:43.116 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:25:43.122 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:43.146 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:43.144+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:43.247 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:43.248 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:43.248 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:43.246+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:43.257 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:25:43.302 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:43.323 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:43.344 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:43.348 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:43.347+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:43.365 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:43.378 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:43.379 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:43.379 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:43.379 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:43.379 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:43.386 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:43.388 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:25:43.414 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:43.429 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:43.449 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:43.449 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:43.448+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:43.474 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:43.491 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:43.512 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:43.515 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:43.515 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:43.515 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:43.515 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:43.515 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.515 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:43.516 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:43.516 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 49.662556438999999, 2020-03-09T00:25:43.516 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999, 2020-03-09T00:25:43.516 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.516 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.516 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.516 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:43.517 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.517 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.517 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.517 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.517 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:43.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017622699999999999 2020-03-09T00:25:43.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.518 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.518 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850137+0000", 2020-03-09T00:25:43.518 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1974000000000003e-05 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850169+0000", 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4702000000000002e-05 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.519 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850193+0000", 2020-03-09T00:25:43.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5790000000000004e-06 2020-03-09T00:25:43.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.520 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.520 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850202+0000", 2020-03-09T00:25:43.520 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013641599999999999 2020-03-09T00:25:43.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.521 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.521 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850338+0000", 2020-03-09T00:25:43.521 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4691999999999999e-05 2020-03-09T00:25:43.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850353+0000", 2020-03-09T00:25:43.522 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011849303 2020-03-09T00:25:43.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.522 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.522 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862202+0000", 2020-03-09T00:25:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1013000000000001e-05 2020-03-09T00:25:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862253+0000", 2020-03-09T00:25:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999 2020-03-09T00:25:43.523 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=37, reqid=client.4751.0:243, at_version=44'17, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:43.524 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 49.656361035000003, 2020-03-09T00:25:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595, 2020-03-09T00:25:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856156+0000", 2020-03-09T00:25:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107948 2020-03-09T00:25:43.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856264+0000", 2020-03-09T00:25:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4422000000000001e-05 2020-03-09T00:25:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.527 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856288+0000", 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5265e-05 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856304+0000", 2020-03-09T00:25:43.528 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3209999999999994e-06 2020-03-09T00:25:43.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.529 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.529 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856313+0000", 2020-03-09T00:25:43.529 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00056587900000000003 2020-03-09T00:25:43.529 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.529 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.530 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.530 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856879+0000", 2020-03-09T00:25:43.530 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1976e-05 2020-03-09T00:25:43.530 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.530 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856891+0000", 2020-03-09T00:25:43.531 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098355160000000007 2020-03-09T00:25:43.531 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.531 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.531 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.531 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866726+0000", 2020-03-09T00:25:43.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6267999999999996e-05 2020-03-09T00:25:43.532 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.532 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.532 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.532 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.866802+0000", 2020-03-09T00:25:43.532 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010646595 2020-03-09T00:25:43.532 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.533 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.533 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.533 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.533 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.533 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(osd.10.23:4 3.3s0 3:c607ec57:::smithi10428818-30:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e48)", 2020-03-09T00:25:43.533 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:43.534 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.723387813999999, 2020-03-09T00:25:43.534 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999, 2020-03-09T00:25:43.534 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.534 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:43.534 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.534 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.535 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:43.535 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.535 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.535 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.535 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:43.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8078999999999999e-05 2020-03-09T00:25:43.536 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.536 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.536 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.536 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789167+0000", 2020-03-09T00:25:43.536 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1499999999999998e-06 2020-03-09T00:25:43.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.537 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.537 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789172+0000", 2020-03-09T00:25:43.537 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6209999999999993e-06 2020-03-09T00:25:43.537 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.537 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789181+0000", 2020-03-09T00:25:43.538 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4939999999999997e-06 2020-03-09T00:25:43.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.538 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.538 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.538 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789185+0000", 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5793e-05 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789231+0000", 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7570000000000002e-06 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.539 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:43.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789235+0000", 2020-03-09T00:25:43.540 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.99423119699999996 2020-03-09T00:25:43.540 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.540 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.540 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.540 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783466+0000", 2020-03-09T00:25:43.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000227851 2020-03-09T00:25:43.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.541 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:43.541 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783694+0000", 2020-03-09T00:25:43.541 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056076839999999999 2020-03-09T00:25:43.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.542 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.542 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789302+0000", 2020-03-09T00:25:43.542 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999 2020-03-09T00:25:43.542 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.542 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.542 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.543 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 53/52 ECSubWriteReply(tid=1, last_complete=34'2, committed=1, applied=1))", 2020-03-09T00:25:43.543 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:43.543 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.719218679999997, 2020-03-09T00:25:43.543 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215, 2020-03-09T00:25:43.544 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.544 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.544 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.544 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.544 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:43.544 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.545 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.545 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:43.545 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9154999999999999e-05 2020-03-09T00:25:43.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.546 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.546 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793357+0000", 2020-03-09T00:25:43.546 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4100000000000008e-06 2020-03-09T00:25:43.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793366+0000", 2020-03-09T00:25:43.547 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3783e-05 2020-03-09T00:25:43.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.547 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.547 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793380+0000", 2020-03-09T00:25:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6469999999999998e-06 2020-03-09T00:25:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793387+0000", 2020-03-09T00:25:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012002081 2020-03-09T00:25:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805389+0000", 2020-03-09T00:25:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0139000000000001e-05 2020-03-09T00:25:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.550 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805409+0000", 2020-03-09T00:25:43.550 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215 2020-03-09T00:25:43.550 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.550 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.550 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.551 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 53/52 ECSubWrite(tid=47, reqid=client.4751.0:355, at_version=53'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:43.551 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:43.551 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.661333935999998, 2020-03-09T00:25:43.551 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252, 2020-03-09T00:25:43.551 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.551 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.551 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.552 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:43.552 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.552 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00081411500000000002 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851997+0000", 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0162999999999999e-05 2020-03-09T00:25:43.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.554 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852027+0000", 2020-03-09T00:25:43.554 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7360999999999999e-05 2020-03-09T00:25:43.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.554 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852045+0000", 2020-03-09T00:25:43.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2069000000000001e-05 2020-03-09T00:25:43.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.555 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.555 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852057+0000", 2020-03-09T00:25:43.555 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000131192 2020-03-09T00:25:43.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.556 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.556 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852188+0000", 2020-03-09T00:25:43.556 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.517e-05 2020-03-09T00:25:43.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852203+0000", 2020-03-09T00:25:43.557 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099256910000000004 2020-03-09T00:25:43.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.557 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.557 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862129+0000", 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6758999999999997e-05 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862176+0000", 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.558 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.559 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=159, reqid=osd.12.0:159, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:43.559 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:43.559 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.692202579000003, 2020-03-09T00:25:43.559 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001, 2020-03-09T00:25:43.560 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.560 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.560 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.560 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.560 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:43.561 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.561 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:43.562 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1613999999999997e-05 2020-03-09T00:25:43.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.562 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.562 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820376+0000", 2020-03-09T00:25:43.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6792e-05 2020-03-09T00:25:43.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.563 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.563 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820403+0000", 2020-03-09T00:25:43.563 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4728e-05 2020-03-09T00:25:43.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.564 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.564 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820417+0000", 2020-03-09T00:25:43.564 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7910000000000005e-06 2020-03-09T00:25:43.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820425+0000", 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010933589999999999 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821519+0000", 2020-03-09T00:25:43.565 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3885e-05 2020-03-09T00:25:43.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.566 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.566 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821533+0000", 2020-03-09T00:25:43.566 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011394939999999999 2020-03-09T00:25:43.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.567 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.567 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832927+0000", 2020-03-09T00:25:43.567 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6371e-05 2020-03-09T00:25:43.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.567 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832974+0000", 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.568 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.569 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=5, reqid=client.4751.0:485, at_version=59'26, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:43.569 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:43.569 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.691560643000003, 2020-03-09T00:25:43.569 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083, 2020-03-09T00:25:43.569 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.569 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.570 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.570 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.570 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.570 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:43.570 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.570 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.571 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.571 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.571 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:43.571 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4487000000000002e-05 2020-03-09T00:25:43.571 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.572 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.572 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821011+0000", 2020-03-09T00:25:43.572 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9616e-05 2020-03-09T00:25:43.572 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.572 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.573 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.573 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821030+0000", 2020-03-09T00:25:43.573 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4766000000000001e-05 2020-03-09T00:25:43.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.574 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.574 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.574 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821045+0000", 2020-03-09T00:25:43.574 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.789e-06 2020-03-09T00:25:43.574 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.575 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821051+0000", 2020-03-09T00:25:43.575 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010220629999999999 2020-03-09T00:25:43.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.575 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.575 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.576 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822073+0000", 2020-03-09T00:25:43.576 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1511999999999999e-05 2020-03-09T00:25:43.576 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.576 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.576 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822085+0000", 2020-03-09T00:25:43.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010894232 2020-03-09T00:25:43.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.577 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.577 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.577 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832979+0000", 2020-03-09T00:25:43.577 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:43.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.578 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.578 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.578 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833022+0000", 2020-03-09T00:25:43.578 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083 2020-03-09T00:25:43.578 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.578 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.579 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.579 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.579 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=6, reqid=client.4751.0:486, at_version=59'27, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:43.579 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:43.579 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.691415761000002, 2020-03-09T00:25:43.579 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978, 2020-03-09T00:25:43.580 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.580 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.580 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.580 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.580 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.580 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:43.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.581 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.581 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.581 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:43.581 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4271000000000003e-05 2020-03-09T00:25:43.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.582 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821155+0000", 2020-03-09T00:25:43.582 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8627e-05 2020-03-09T00:25:43.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.582 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.582 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.583 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821174+0000", 2020-03-09T00:25:43.583 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4825e-05 2020-03-09T00:25:43.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.585 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.585 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821189+0000", 2020-03-09T00:25:43.585 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4070000000000001e-06 2020-03-09T00:25:43.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.585 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.585 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821194+0000", 2020-03-09T00:25:43.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039590839999999999 2020-03-09T00:25:43.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.586 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.586 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825153+0000", 2020-03-09T00:25:43.586 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3074999999999999e-05 2020-03-09T00:25:43.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.587 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825166+0000", 2020-03-09T00:25:43.587 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078593580000000003 2020-03-09T00:25:43.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.587 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833026+0000", 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2330999999999999e-05 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833058+0000", 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978 2020-03-09T00:25:43.588 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.589 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.589 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.589 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=59'43, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:43.589 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:43.589 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.691210738999999, 2020-03-09T00:25:43.590 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525, 2020-03-09T00:25:43.590 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.590 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.590 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.590 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.590 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.590 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6616999999999997e-05 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.591 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821363+0000", 2020-03-09T00:25:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1061e-05 2020-03-09T00:25:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821384+0000", 2020-03-09T00:25:43.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5315000000000002e-05 2020-03-09T00:25:43.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.593 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.593 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.593 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821399+0000", 2020-03-09T00:25:43.593 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0929999999999999e-06 2020-03-09T00:25:43.593 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.594 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821405+0000", 2020-03-09T00:25:43.594 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003024559 2020-03-09T00:25:43.594 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.594 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.594 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824430+0000", 2020-03-09T00:25:43.595 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2744e-05 2020-03-09T00:25:43.595 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.595 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.595 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.595 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824443+0000", 2020-03-09T00:25:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010025054 2020-03-09T00:25:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834468+0000", 2020-03-09T00:25:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0082e-05 2020-03-09T00:25:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834508+0000", 2020-03-09T00:25:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525 2020-03-09T00:25:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=7, reqid=client.4751.0:487, at_version=59'28, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.672868147999999, 2020-03-09T00:25:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916, 2020-03-09T00:25:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7596999999999998e-05 2020-03-09T00:25:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839706+0000", 2020-03-09T00:25:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5558000000000002e-05 2020-03-09T00:25:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839732+0000", 2020-03-09T00:25:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0251999999999999e-05 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839752+0000", 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0780000000000002e-06 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839761+0000", 2020-03-09T00:25:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010019800000000001 2020-03-09T00:25:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839861+0000", 2020-03-09T00:25:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9032e-05 2020-03-09T00:25:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839881+0000", 2020-03-09T00:25:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014066339000000001 2020-03-09T00:25:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853947+0000", 2020-03-09T00:25:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000210862 2020-03-09T00:25:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854158+0000", 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=164, reqid=client.4751.0:490, at_version=59'45, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.672649716999999, 2020-03-09T00:25:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001, 2020-03-09T00:25:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000195997 2020-03-09T00:25:43.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.610 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.610 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840063+0000", 2020-03-09T00:25:43.610 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7005e-05 2020-03-09T00:25:43.610 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.610 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840090+0000", 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6943999999999999e-05 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840107+0000", 2020-03-09T00:25:43.611 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7489999999999998e-06 2020-03-09T00:25:43.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.612 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.612 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840115+0000", 2020-03-09T00:25:43.612 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019725999999999999 2020-03-09T00:25:43.612 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.612 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840312+0000", 2020-03-09T00:25:43.613 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4409e-05 2020-03-09T00:25:43.613 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.613 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.613 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.613 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840327+0000", 2020-03-09T00:25:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018701885000000001 2020-03-09T00:25:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859028+0000", 2020-03-09T00:25:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7057e-05 2020-03-09T00:25:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859076+0000", 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.616 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:43.616 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:43.616 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.669140937000002, 2020-03-09T00:25:43.616 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002, 2020-03-09T00:25:43.616 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.616 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.617 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.617 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.617 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:43.617 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:43.618 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012568360000000001 2020-03-09T00:25:43.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.618 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.618 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844633+0000", 2020-03-09T00:25:43.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0247999999999999e-05 2020-03-09T00:25:43.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.619 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.619 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844663+0000", 2020-03-09T00:25:43.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6184e-05 2020-03-09T00:25:43.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.620 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844679+0000", 2020-03-09T00:25:43.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8560000000000001e-06 2020-03-09T00:25:43.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.620 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844686+0000", 2020-03-09T00:25:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8074e-05 2020-03-09T00:25:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844744+0000", 2020-03-09T00:25:43.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9119000000000001e-05 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844763+0000", 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014486196999999999 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859250+0000", 2020-03-09T00:25:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115911 2020-03-09T00:25:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859365+0000", 2020-03-09T00:25:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002 2020-03-09T00:25:43.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.624 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.624 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=90, reqid=client.4751.0:499, at_version=59'56, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:43.624 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 34.663435165000003, 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999, 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.625 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:43.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:43.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050415699999999998 2020-03-09T00:25:43.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849586+0000", 2020-03-09T00:25:43.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9476e-05 2020-03-09T00:25:43.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849615+0000", 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8382000000000001e-05 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.628 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849634+0000", 2020-03-09T00:25:43.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0439999999999996e-06 2020-03-09T00:25:43.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849642+0000", 2020-03-09T00:25:43.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018423140000000001 2020-03-09T00:25:43.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.630 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.630 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851484+0000", 2020-03-09T00:25:43.630 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3304e-05 2020-03-09T00:25:43.630 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.630 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851497+0000", 2020-03-09T00:25:43.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097988120000000005 2020-03-09T00:25:43.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861296+0000", 2020-03-09T00:25:43.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1790999999999999e-05 2020-03-09T00:25:43.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861348+0000", 2020-03-09T00:25:43.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999 2020-03-09T00:25:43.632 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.632 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.633 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.633 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:607 3.3s0 3:c2119ac7:::smithi10428818-4:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:43.633 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:43.633 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.710966975000002, 2020-03-09T00:25:43.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011089972999999999, 2020-03-09T00:25:43.634 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.634 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:43.634 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:43.634 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:43.634 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:43.634 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 607 2020-03-09T00:25:43.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.635 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:43.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.636 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.636 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:43.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7959e-05 2020-03-09T00:25:43.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.637 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801588+0000", 2020-03-09T00:25:43.637 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.676e-06 2020-03-09T00:25:43.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.637 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.637 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801592+0000", 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3799999999999999e-06 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801598+0000", 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5610000000000004e-06 2020-03-09T00:25:43.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801605+0000", 2020-03-09T00:25:43.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015221259999999999 2020-03-09T00:25:43.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.803127+0000", 2020-03-09T00:25:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012211399999999999 2020-03-09T00:25:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.803249+0000", 2020-03-09T00:25:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0093911570000000007 2020-03-09T00:25:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.642 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.642 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.812640+0000", 2020-03-09T00:25:43.642 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011089972999999999 2020-03-09T00:25:43.642 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.642 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.642 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.643 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:608 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:43.643 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:43.643 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.710881739000001, 2020-03-09T00:25:43.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013652041, 2020-03-09T00:25:43.643 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.644 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:43.644 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:43.644 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:43.644 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:43.644 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 608 2020-03-09T00:25:43.644 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.644 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.645 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.645 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:43.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5256999999999997e-05 2020-03-09T00:25:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801680+0000", 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3590000000000002e-06 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801688+0000", 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1552999999999999e-05 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801699+0000", 2020-03-09T00:25:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3519999999999997e-06 2020-03-09T00:25:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801705+0000", 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003598459 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.805303+0000", 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000173425 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.805477+0000", 2020-03-09T00:25:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098106359999999993 2020-03-09T00:25:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.815287+0000", 2020-03-09T00:25:43.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013652041 2020-03-09T00:25:43.651 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.651 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.651 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.651 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:609 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:43.652 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:43.652 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.710762151000001, 2020-03-09T00:25:43.652 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013941073999999999, 2020-03-09T00:25:43.652 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.652 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:43.652 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:43.652 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:43.653 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:43.653 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 609 2020-03-09T00:25:43.653 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.653 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.653 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.653 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.653 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:43.654 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.654 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.654 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.654 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.654 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:43.654 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0500000000000006e-05 2020-03-09T00:25:43.654 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801825+0000", 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.614e-06 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.655 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801831+0000", 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5819999999999998e-06 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801837+0000", 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.619e-06 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801841+0000", 2020-03-09T00:25:43.657 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042523969999999998 2020-03-09T00:25:43.657 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806094+0000", 2020-03-09T00:25:43.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011631799999999999 2020-03-09T00:25:43.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.658 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:43.658 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806210+0000", 2020-03-09T00:25:43.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094860440000000008 2020-03-09T00:25:43.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.659 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.659 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.815696+0000", 2020-03-09T00:25:43.659 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013941073999999999 2020-03-09T00:25:43.659 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.659 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.659 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.660 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.660 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:610 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:43.660 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:43.660 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.710624733, 2020-03-09T00:25:43.660 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014287645, 2020-03-09T00:25:43.660 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.661 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:43.661 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:43.661 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:43.661 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:43.661 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 610 2020-03-09T00:25:43.661 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.662 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.662 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.662 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.662 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:43.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.663 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.663 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:43.663 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5413000000000005e-05 2020-03-09T00:25:43.663 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.663 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801968+0000", 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4571e-05 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801982+0000", 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3568e-05 2020-03-09T00:25:43.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.665 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.665 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.665 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801996+0000", 2020-03-09T00:25:43.665 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4940000000000003e-06 2020-03-09T00:25:43.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.665 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.665 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.666 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.802001+0000", 2020-03-09T00:25:43.666 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046307759999999996 2020-03-09T00:25:43.666 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.666 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.666 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.666 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806632+0000", 2020-03-09T00:25:43.666 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011677599999999999 2020-03-09T00:25:43.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.667 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:43.667 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806749+0000", 2020-03-09T00:25:43.667 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094310469999999997 2020-03-09T00:25:43.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.816180+0000", 2020-03-09T00:25:43.668 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014287645 2020-03-09T00:25:43.668 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.668 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.668 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.669 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.4s6 75/67 ECSubWrite(tid=93, reqid=client.4751.0:637, at_version=75'8, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:43.669 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:43.669 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 18.702749572999998, 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021185215, 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:43.670 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.671 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.671 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:43.671 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021834200000000002 2020-03-09T00:25:43.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811951+0000", 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8116999999999999e-05 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811979+0000", 2020-03-09T00:25:43.672 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3502e-05 2020-03-09T00:25:43.673 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.673 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.673 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.673 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811992+0000", 2020-03-09T00:25:43.673 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9779999999999999e-06 2020-03-09T00:25:43.673 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.674 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.674 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.674 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811999+0000", 2020-03-09T00:25:43.674 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6839999999999998e-05 2020-03-09T00:25:43.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.674 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.674 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.675 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.812056+0000", 2020-03-09T00:25:43.675 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5362e-05 2020-03-09T00:25:43.675 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.675 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.676 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.812072+0000", 2020-03-09T00:25:43.676 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01883961 2020-03-09T00:25:43.676 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.830911+0000", 2020-03-09T00:25:43.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1386e-05 2020-03-09T00:25:43.677 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.677 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.677 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.830953+0000", 2020-03-09T00:25:43.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021185215 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.6s1 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.820270+0000", 2020-03-09T00:25:43.678 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 13.692246453999999, 2020-03-09T00:25:43.679 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013028451, 2020-03-09T00:25:43.679 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.679 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.679 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.679 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.679 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.679 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.820270+0000", 2020-03-09T00:25:43.680 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.680 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.680 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.680 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.680 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.820270+0000", 2020-03-09T00:25:43.680 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001056408 2020-03-09T00:25:43.680 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.681 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.681 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.681 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821327+0000", 2020-03-09T00:25:43.681 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2941000000000002e-05 2020-03-09T00:25:43.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.681 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.681 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821360+0000", 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4844999999999999e-05 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821375+0000", 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9319999999999999e-06 2020-03-09T00:25:43.682 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.683 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.683 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.683 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821382+0000", 2020-03-09T00:25:43.683 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000103878 2020-03-09T00:25:43.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.683 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.683 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.684 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821485+0000", 2020-03-09T00:25:43.684 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2746e-05 2020-03-09T00:25:43.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.684 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.684 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.684 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821498+0000", 2020-03-09T00:25:43.684 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011737274000000001 2020-03-09T00:25:43.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.685 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.685 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.685 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.833236+0000", 2020-03-09T00:25:43.685 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3427000000000001e-05 2020-03-09T00:25:43.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.685 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.686 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.686 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.833299+0000", 2020-03-09T00:25:43.686 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013028451 2020-03-09T00:25:43.686 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.686 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.686 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.687 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.687 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:43.687 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.884724+0000", 2020-03-09T00:25:43.687 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 8.6277931589999994, 2020-03-09T00:25:43.687 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019060613000000001, 2020-03-09T00:25:43.687 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:43.688 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:43.688 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:43.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:43.689 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884724+0000", 2020-03-09T00:25:43.689 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884724+0000", 2020-03-09T00:25:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022766119999999999 2020-03-09T00:25:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:43.692 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887000+0000", 2020-03-09T00:25:43.692 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1268999999999999e-05 2020-03-09T00:25:43.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:43.693 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887022+0000", 2020-03-09T00:25:43.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2214e-05 2020-03-09T00:25:43.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.693 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.693 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:43.693 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887034+0000", 2020-03-09T00:25:43.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5749999999999997e-06 2020-03-09T00:25:43.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.694 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.694 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:43.694 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887040+0000", 2020-03-09T00:25:43.694 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001634417 2020-03-09T00:25:43.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.694 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.888675+0000", 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5466999999999999e-05 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.888690+0000", 2020-03-09T00:25:43.695 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015039879000000001 2020-03-09T00:25:43.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.696 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:43.696 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.903730+0000", 2020-03-09T00:25:43.696 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.418e-05 2020-03-09T00:25:43.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:43.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:43.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:43.697 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.903784+0000", 2020-03-09T00:25:43.697 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019060613000000001 2020-03-09T00:25:43.697 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.697 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.697 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.697 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:43.698 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:43.698 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:43.700 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:25:43.701 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:43.701 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:43.701 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:43.698+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:43.774 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:43.798 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:43.799 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.800 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.800 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.800 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.800 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.801 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.801 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:43.801 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:43.802 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.802 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.802 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.802 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.802 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.803 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.803 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.803 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.803 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:43.807 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:43.805+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:43.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:43.829 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:43.830 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:43.830 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:43.830 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:43.839 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:25:43.864 INFO:tasks.rados.rados.0.smithi104.stdout:359: done (1 left) 2020-03-09T00:25:43.864 INFO:tasks.rados.rados.0.smithi104.stdout:362: done (0 left) 2020-03-09T00:25:43.864 INFO:tasks.rados.rados.0.smithi104.stdout:363: read oid 4 snap 27 2020-03-09T00:25:43.864 INFO:tasks.rados.rados.0.smithi104.stdout:363: expect (ObjNum 34 snap 0 seq_num 34) 2020-03-09T00:25:43.864 INFO:tasks.rados.rados.0.smithi104.stdout:364: read oid 37 snap -1 2020-03-09T00:25:43.865 INFO:tasks.rados.rados.0.smithi104.stdout:364: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:43.865 INFO:tasks.rados.rados.0.smithi104.stdout:365: rmattr oid 16 current snap is 30 2020-03-09T00:25:43.865 INFO:tasks.rados.rados.0.smithi104.stdout:365: done (2 left) 2020-03-09T00:25:43.865 INFO:tasks.rados.rados.0.smithi104.stdout:366: setattr oid 50 current snap is 30 2020-03-09T00:25:43.866 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 28 current snap is 30 2020-03-09T00:25:43.866 INFO:tasks.rados.rados.0.smithi104.stdout:367: seq_num 143 ranges {2146304=458752,2605056=475136,3080192=655360,3735552=671744,4407296=147456} 2020-03-09T00:25:43.880 INFO:tasks.rados.rados.0.smithi104.stdout:367: writing smithi10428818-28 from 2146304 to 2605056 tid 1 2020-03-09T00:25:43.887 INFO:tasks.rados.rados.0.smithi104.stdout:367: writing smithi10428818-28 from 2605056 to 3080192 tid 2 2020-03-09T00:25:43.892 INFO:tasks.rados.rados.0.smithi104.stdout:367: writing smithi10428818-28 from 3080192 to 3735552 tid 3 2020-03-09T00:25:43.897 INFO:tasks.rados.rados.0.smithi104.stdout:367: writing smithi10428818-28 from 3735552 to 4407296 tid 4 2020-03-09T00:25:43.898 INFO:tasks.rados.rados.0.smithi104.stdout:367: writing smithi10428818-28 from 4407296 to 4554752 tid 5 2020-03-09T00:25:43.898 INFO:tasks.rados.rados.0.smithi104.stdout:368: delete oid 46 current snap is 30 2020-03-09T00:25:43.900 INFO:tasks.rados.rados.0.smithi104.stdout:364: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:43.925 INFO:tasks.rados.rados.0.smithi104.stdout:363: expect (ObjNum 34 snap 0 seq_num 34) 2020-03-09T00:25:43.926 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:43.924+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:43.935 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 50 v 54 (ObjNum 1 snap 32510 seq_num 3968685776) dirty exists 2020-03-09T00:25:43.935 INFO:tasks.rados.rados.0.smithi104.stdout:367: finishing write tid 1 to smithi10428818-28 2020-03-09T00:25:43.935 INFO:tasks.rados.rados.0.smithi104.stdout:367: finishing write tid 2 to smithi10428818-28 2020-03-09T00:25:43.935 INFO:tasks.rados.rados.0.smithi104.stdout:363: done (4 left) 2020-03-09T00:25:43.935 INFO:tasks.rados.rados.0.smithi104.stdout:364: done (3 left) 2020-03-09T00:25:43.935 INFO:tasks.rados.rados.0.smithi104.stdout:366: done (2 left) 2020-03-09T00:25:43.935 INFO:tasks.rados.rados.0.smithi104.stdout:368: done (1 left) 2020-03-09T00:25:43.936 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 36 current snap is 30 2020-03-09T00:25:43.936 INFO:tasks.rados.rados.0.smithi104.stdout:369: seq_num 145 ranges {3014656=32768} 2020-03-09T00:25:43.951 INFO:tasks.rados.rados.0.smithi104.stdout:369: writing smithi10428818-36 from 3014656 to 3047424 tid 1 2020-03-09T00:25:43.951 INFO:tasks.rados.rados.0.smithi104.stdout:370: rollback oid 8 current snap is 30 2020-03-09T00:25:43.951 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 8 to 27 2020-03-09T00:25:43.952 INFO:tasks.rados.rados.0.smithi104.stdout:371: rollback oid 12 current snap is 30 2020-03-09T00:25:43.952 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 12 to 28 2020-03-09T00:25:43.952 INFO:tasks.rados.rados.0.smithi104.stdout:372: read oid 37 snap -1 2020-03-09T00:25:43.952 INFO:tasks.rados.rados.0.smithi104.stdout:372: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:43.952 INFO:tasks.rados.rados.0.smithi104.stdout:373: copy_from oid 41 from oid 34 current snap is 30 2020-03-09T00:25:43.952 INFO:tasks.rados.rados.0.smithi104.stdout:374: setattr oid 23 current snap is 30 2020-03-09T00:25:43.953 INFO:tasks.rados.rados.0.smithi104.stdout:375: rollback oid 22 current snap is 30 2020-03-09T00:25:43.953 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 22 to 28 2020-03-09T00:25:43.953 INFO:tasks.rados.rados.0.smithi104.stdout:376: snap_create 2020-03-09T00:25:43.959 INFO:tasks.rados.rados.0.smithi104.stdout:372: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:43.981 INFO:tasks.rados.rados.0.smithi104.stdout:370: finishing rollback tid 0 to smithi10428818-8 2020-03-09T00:25:43.981 INFO:tasks.rados.rados.0.smithi104.stdout:370: finishing rollback tid 1 to smithi10428818-8 2020-03-09T00:25:43.981 INFO:tasks.rados.rados.0.smithi104.stdout:369: finishing write tid 1 to smithi10428818-36 2020-03-09T00:25:43.982 INFO:tasks.rados.rados.0.smithi104.stdout:369: finishing write tid 3 to smithi10428818-36 2020-03-09T00:25:43.982 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 23 v 61 (ObjNum 134 snap 26 seq_num 134) dirty exists 2020-03-09T00:25:43.982 INFO:tasks.rados.rados.0.smithi104.stdout:370: finishing rollback tid 2 to smithi10428818-8 2020-03-09T00:25:43.982 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 8 v 39 (ObjNum 74 snap 11 seq_num 74) dirty exists 2020-03-09T00:25:43.982 INFO:tasks.rados.rados.0.smithi104.stdout:371: finishing rollback tid 0 to smithi10428818-12 2020-03-09T00:25:43.982 INFO:tasks.rados.rados.0.smithi104.stdout:373: finishing copy_from to smithi10428818-41 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:373: got expected ENOENT (src dne) 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:373: finishing copy_from racing read to smithi10428818-41 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:369: finishing write tid 4 to smithi10428818-36 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 36 v 37 (ObjNum 145 snap 30 seq_num 145) dirty exists 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:369: left oid 36 (ObjNum 145 snap 30 seq_num 145) 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:371: finishing rollback tid 1 to smithi10428818-12 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:371: finishing rollback tid 2 to smithi10428818-12 2020-03-09T00:25:43.983 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 12 v 115 (ObjNum 75 snap 11 seq_num 75) dirty exists 2020-03-09T00:25:43.984 INFO:tasks.rados.rados.0.smithi104.stdout:375: finishing rollback tid 0 to smithi10428818-22 2020-03-09T00:25:43.984 INFO:tasks.rados.rados.0.smithi104.stdout:375: finishing rollback tid 1 to smithi10428818-22 2020-03-09T00:25:43.984 INFO:tasks.rados.rados.0.smithi104.stdout:375: finishing rollback tid 2 to smithi10428818-22 2020-03-09T00:25:43.984 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 22 v 174 (ObjNum 124 snap 24 seq_num 124) dirty exists 2020-03-09T00:25:43.984 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:43.985 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:43.985 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:43.993 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:25:44.025 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:44.024+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:44.105 INFO:tasks.rados.rados.0.smithi104.stdout:367: finishing write tid 3 to smithi10428818-28 2020-03-09T00:25:44.106 INFO:tasks.rados.rados.0.smithi104.stdout:367: finishing write tid 4 to smithi10428818-28 2020-03-09T00:25:44.106 INFO:tasks.rados.rados.0.smithi104.stdout:367: finishing write tid 5 to smithi10428818-28 2020-03-09T00:25:44.106 INFO:tasks.rados.rados.0.smithi104.stdout:367: finishing write tid 7 to smithi10428818-28 2020-03-09T00:25:44.108 INFO:tasks.rados.rados.0.smithi104.stdout:367: finishing write tid 8 to smithi10428818-28 2020-03-09T00:25:44.108 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 28 v 66 (ObjNum 143 snap 30 seq_num 143) dirty exists 2020-03-09T00:25:44.108 INFO:tasks.rados.rados.0.smithi104.stdout:367: left oid 28 (ObjNum 143 snap 30 seq_num 143) 2020-03-09T00:25:44.126 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:44.125+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:44.152 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:44.153 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:44.153 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:44.153 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:44.154 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.154 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:207 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 5] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:44.154 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:44.154 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 56.292718530999998, 2020-03-09T00:25:44.154 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001, 2020-03-09T00:25:44.155 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.155 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.155 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.156 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.157 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.157 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:44.157 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7176999999999996e-05 2020-03-09T00:25:44.157 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.157 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.158 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.158 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856995+0000", 2020-03-09T00:25:44.158 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7161999999999999e-05 2020-03-09T00:25:44.158 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.158 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857013+0000", 2020-03-09T00:25:44.159 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2879e-05 2020-03-09T00:25:44.159 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.159 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.159 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.159 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857026+0000", 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6669999999999996e-06 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857033+0000", 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6767999999999997e-05 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.160 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857080+0000", 2020-03-09T00:25:44.161 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4165999999999994e-05 2020-03-09T00:25:44.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.161 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.161 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.161 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857154+0000", 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068311889999999997 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.863985+0000", 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041278249999999999 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.162 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868113+0000", 2020-03-09T00:25:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001971132 2020-03-09T00:25:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.163 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870084+0000", 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113266 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870198+0000", 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.164 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:208 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 56.292548785000001, 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001, 2020-03-09T00:25:44.165 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.166 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:44.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.166 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.166 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 208 2020-03-09T00:25:44.166 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.166 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.167 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.167 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:44.167 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.167 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.167 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:44.168 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5717000000000003e-05 2020-03-09T00:25:44.168 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.168 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.168 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.168 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857134+0000", 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3659999999999997e-06 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857140+0000", 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2546e-05 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.169 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857153+0000", 2020-03-09T00:25:44.170 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7190000000000001e-06 2020-03-09T00:25:44.170 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.170 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.170 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.170 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857157+0000", 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134509 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857292+0000", 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5519999999999999e-05 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.171 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:44.172 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857327+0000", 2020-03-09T00:25:44.172 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010666132 2020-03-09T00:25:44.172 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.172 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.172 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867994+0000", 2020-03-09T00:25:44.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110682 2020-03-09T00:25:44.173 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.173 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.173 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:44.173 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868104+0000", 2020-03-09T00:25:44.173 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00321515 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871319+0000", 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011457 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.174 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871434+0000", 2020-03-09T00:25:44.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013121699999999999 2020-03-09T00:25:44.175 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.175 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.175 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.175 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871565+0000", 2020-03-09T00:25:44.175 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 44/32 ECSubWrite(tid=44, reqid=osd.13.0:44, at_version=44'11, trim_to=0'0, roll_forward_to=42'9))", 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:44.176 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 50.300495554000001, 2020-03-09T00:25:44.177 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002, 2020-03-09T00:25:44.177 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.177 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.177 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.177 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.177 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.177 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849141+0000", 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8740999999999999e-05 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.178 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849200+0000", 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2819e-05 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849223+0000", 2020-03-09T00:25:44.179 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5366000000000002e-05 2020-03-09T00:25:44.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.180 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.180 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849238+0000", 2020-03-09T00:25:44.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7410000000000003e-06 2020-03-09T00:25:44.180 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849245+0000", 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5445999999999996e-05 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849310+0000", 2020-03-09T00:25:44.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3651e-05 2020-03-09T00:25:44.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.182 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.182 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.849324+0000", 2020-03-09T00:25:44.182 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0057096769999999998 2020-03-09T00:25:44.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855034+0000", 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001136 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.855147+0000", 2020-03-09T00:25:44.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060060410000000002 2020-03-09T00:25:44.184 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.184 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.184 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.184 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:44.184 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:44.185 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 50.299353969000002, 2020-03-09T00:25:44.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998, 2020-03-09T00:25:44.185 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.185 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.185 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:44.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:44.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018406799999999999 2020-03-09T00:25:44.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850467+0000", 2020-03-09T00:25:44.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2490999999999999e-05 2020-03-09T00:25:44.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850489+0000", 2020-03-09T00:25:44.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6807999999999999e-05 2020-03-09T00:25:44.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:44.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9419999999999995e-06 2020-03-09T00:25:44.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850514+0000", 2020-03-09T00:25:44.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088484100000000003 2020-03-09T00:25:44.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851399+0000", 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2279999999999999e-05 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851411+0000", 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083268249999999995 2020-03-09T00:25:44.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.192 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859738+0000", 2020-03-09T00:25:44.192 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2222999999999998e-05 2020-03-09T00:25:44.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.192 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.193 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859790+0000", 2020-03-09T00:25:44.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998 2020-03-09T00:25:44.193 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.193 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.193 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.193 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.193 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 45.332239281, 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001, 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:44.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069548899999999998 2020-03-09T00:25:44.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818093+0000", 2020-03-09T00:25:44.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6591999999999999e-05 2020-03-09T00:25:44.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818120+0000", 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5192e-05 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818135+0000", 2020-03-09T00:25:44.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4989999999999997e-06 2020-03-09T00:25:44.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818142+0000", 2020-03-09T00:25:44.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4997000000000002e-05 2020-03-09T00:25:44.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818197+0000", 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3164e-05 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818210+0000", 2020-03-09T00:25:44.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092824440000000008 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827493+0000", 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2645999999999997e-05 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827546+0000", 2020-03-09T00:25:44.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001 2020-03-09T00:25:44.201 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.201 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.201 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s3 53/48 ECSubWrite(tid=30, reqid=client.4751.0:346, at_version=53'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.322740946000003, 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059893999999999998, 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:44.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.203 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826896+0000", 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000139486 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827035+0000", 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5199999999999999e-05 2020-03-09T00:25:44.204 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827061+0000", 2020-03-09T00:25:44.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376999999999999e-05 2020-03-09T00:25:44.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827076+0000", 2020-03-09T00:25:44.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8279999999999997e-06 2020-03-09T00:25:44.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827083+0000", 2020-03-09T00:25:44.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00050597700000000001 2020-03-09T00:25:44.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827589+0000", 2020-03-09T00:25:44.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2581e-05 2020-03-09T00:25:44.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827601+0000", 2020-03-09T00:25:44.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052374839999999997 2020-03-09T00:25:44.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832839+0000", 2020-03-09T00:25:44.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6467000000000001e-05 2020-03-09T00:25:44.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832885+0000", 2020-03-09T00:25:44.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059893999999999998 2020-03-09T00:25:44.210 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.210 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.210 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 54/48 ECSubWrite(tid=8, reqid=client.4751.0:376, at_version=54'17, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 40.340226131000001, 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002, 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.211 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.212 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.212 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.212 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:44.212 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000509736 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809920+0000", 2020-03-09T00:25:44.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8026999999999999e-05 2020-03-09T00:25:44.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809938+0000", 2020-03-09T00:25:44.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829e-05 2020-03-09T00:25:44.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809954+0000", 2020-03-09T00:25:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8799999999999997e-06 2020-03-09T00:25:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809963+0000", 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3013000000000003e-05 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810006+0000", 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2413e-05 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810019+0000", 2020-03-09T00:25:44.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067421620000000003 2020-03-09T00:25:44.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816761+0000", 2020-03-09T00:25:44.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3118999999999997e-05 2020-03-09T00:25:44.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816814+0000", 2020-03-09T00:25:44.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002 2020-03-09T00:25:44.219 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.219 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.220 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.220 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 59/48 ECSubWrite(tid=22, reqid=osd.9.0:22, at_version=59'30, trim_to=0'0, roll_forward_to=58'28))", 2020-03-09T00:25:44.220 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.337032460000003, 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067353739999999997, 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.221 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:44.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812604+0000", 2020-03-09T00:25:44.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4576000000000001e-05 2020-03-09T00:25:44.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812659+0000", 2020-03-09T00:25:44.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0911999999999998e-05 2020-03-09T00:25:44.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812690+0000", 2020-03-09T00:25:44.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4375999999999999e-05 2020-03-09T00:25:44.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812704+0000", 2020-03-09T00:25:44.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2119999999999997e-06 2020-03-09T00:25:44.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812711+0000", 2020-03-09T00:25:44.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016912599999999999 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812881+0000", 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.377e-05 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.812894+0000", 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0064052889999999998 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819300+0000", 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0113e-05 2020-03-09T00:25:44.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819340+0000", 2020-03-09T00:25:44.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067353739999999997 2020-03-09T00:25:44.228 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.228 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.228 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:507 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 54] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.244947383000003, 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001, 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.229 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.230 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 507 2020-03-09T00:25:44.230 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.230 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.230 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:44.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4511999999999997e-05 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904734+0000", 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3980000000000004e-06 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904740+0000", 2020-03-09T00:25:44.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4293e-05 2020-03-09T00:25:44.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904755+0000", 2020-03-09T00:25:44.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4449999999999996e-06 2020-03-09T00:25:44.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904763+0000", 2020-03-09T00:25:44.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0075999999999999e-05 2020-03-09T00:25:44.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904823+0000", 2020-03-09T00:25:44.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8325999999999994e-05 2020-03-09T00:25:44.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904911+0000", 2020-03-09T00:25:44.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091150910000000005 2020-03-09T00:25:44.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914027+0000", 2020-03-09T00:25:44.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097858200000000006 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923812+0000", 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072064440000000002 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931019+0000", 2020-03-09T00:25:44.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6091000000000001e-05 2020-03-09T00:25:44.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931085+0000", 2020-03-09T00:25:44.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:508 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:44.239 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.244835315000003, 2020-03-09T00:25:44.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261, 2020-03-09T00:25:44.240 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.240 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:44.240 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.240 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.240 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.240 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 508 2020-03-09T00:25:44.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.241 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.241 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.241 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.241 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:44.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:44.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4413000000000002e-05 2020-03-09T00:25:44.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904826+0000", 2020-03-09T00:25:44.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1389999999999997e-06 2020-03-09T00:25:44.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904830+0000", 2020-03-09T00:25:44.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3339999999999996e-06 2020-03-09T00:25:44.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904837+0000", 2020-03-09T00:25:44.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8739999999999999e-06 2020-03-09T00:25:44.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904840+0000", 2020-03-09T00:25:44.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000222303 2020-03-09T00:25:44.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905063+0000", 2020-03-09T00:25:44.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5607999999999999e-05 2020-03-09T00:25:44.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:44.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905128+0000", 2020-03-09T00:25:44.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018331841000000001 2020-03-09T00:25:44.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.251 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.251 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923460+0000", 2020-03-09T00:25:44.251 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012596800000000001 2020-03-09T00:25:44.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.252 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.252 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:44.252 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923586+0000", 2020-03-09T00:25:44.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0084087399999999996 2020-03-09T00:25:44.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931995+0000", 2020-03-09T00:25:44.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010665 2020-03-09T00:25:44.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932101+0000", 2020-03-09T00:25:44.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012239099999999999 2020-03-09T00:25:44.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932224+0000", 2020-03-09T00:25:44.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261 2020-03-09T00:25:44.255 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.256 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.256 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.256 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 68/48 ECSubWrite(tid=38, reqid=client.4751.0:589, at_version=68'45, trim_to=0'0, roll_forward_to=67'41))", 2020-03-09T00:25:44.257 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.987799+0000", 2020-03-09T00:25:44.257 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 26.161837583000001, 2020-03-09T00:25:44.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066719780000000003, 2020-03-09T00:25:44.257 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.258 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.258 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.987799+0000", 2020-03-09T00:25:44.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.987799+0000", 2020-03-09T00:25:44.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00034108499999999998 2020-03-09T00:25:44.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988140+0000", 2020-03-09T00:25:44.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9864000000000002e-05 2020-03-09T00:25:44.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988160+0000", 2020-03-09T00:25:44.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5350999999999999e-05 2020-03-09T00:25:44.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988176+0000", 2020-03-09T00:25:44.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8819999999999998e-06 2020-03-09T00:25:44.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.263 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.263 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988181+0000", 2020-03-09T00:25:44.263 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5544999999999999e-05 2020-03-09T00:25:44.264 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.264 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.264 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.264 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988217+0000", 2020-03-09T00:25:44.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.294e-06 2020-03-09T00:25:44.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.988224+0000", 2020-03-09T00:25:44.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061940880000000004 2020-03-09T00:25:44.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.994418+0000", 2020-03-09T00:25:44.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2868999999999998e-05 2020-03-09T00:25:44.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.994471+0000", 2020-03-09T00:25:44.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066719780000000003 2020-03-09T00:25:44.268 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.268 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.269 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.269 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:598 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:44.270 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:44.270 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.361099374999998, 2020-03-09T00:25:44.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014064482999999999, 2020-03-09T00:25:44.270 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.271 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.271 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.271 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 598 2020-03-09T00:25:44.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.272 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:44.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:44.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4369999999999999e-05 2020-03-09T00:25:44.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.275 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788562+0000", 2020-03-09T00:25:44.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8599999999999998e-06 2020-03-09T00:25:44.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.276 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.276 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.276 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788568+0000", 2020-03-09T00:25:44.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5980000000000003e-06 2020-03-09T00:25:44.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.277 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788577+0000", 2020-03-09T00:25:44.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.955e-06 2020-03-09T00:25:44.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788582+0000", 2020-03-09T00:25:44.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5593000000000002e-05 2020-03-09T00:25:44.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788628+0000", 2020-03-09T00:25:44.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3660000000000003e-06 2020-03-09T00:25:44.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:44.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788635+0000", 2020-03-09T00:25:44.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019670759999999999 2020-03-09T00:25:44.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.790602+0000", 2020-03-09T00:25:44.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0575999999999995e-05 2020-03-09T00:25:44.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.790693+0000", 2020-03-09T00:25:44.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00085902600000000004 2020-03-09T00:25:44.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.791552+0000", 2020-03-09T00:25:44.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052594449999999997 2020-03-09T00:25:44.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796811+0000", 2020-03-09T00:25:44.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056778230000000002 2020-03-09T00:25:44.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802489+0000", 2020-03-09T00:25:44.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011279499999999999 2020-03-09T00:25:44.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.287 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.288 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.288 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802602+0000", 2020-03-09T00:25:44.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014064482999999999 2020-03-09T00:25:44.288 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.289 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.290 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:599 3.as0 3:51058371:::smithi10428818-28:head [rollback 1a] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:44.290 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:44.290 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.361010954000001, 2020-03-09T00:25:44.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014184085000000001, 2020-03-09T00:25:44.291 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.291 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.291 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.292 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.292 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.292 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 599 2020-03-09T00:25:44.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.292 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:44.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:44.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8149999999999998e-05 2020-03-09T00:25:44.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788674+0000", 2020-03-09T00:25:44.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7279999999999998e-06 2020-03-09T00:25:44.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788681+0000", 2020-03-09T00:25:44.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.347e-05 2020-03-09T00:25:44.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788694+0000", 2020-03-09T00:25:44.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8910000000000003e-06 2020-03-09T00:25:44.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.299 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788699+0000", 2020-03-09T00:25:44.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2481999999999998e-05 2020-03-09T00:25:44.300 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.300 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.300 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.300 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788732+0000", 2020-03-09T00:25:44.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4740000000000002e-06 2020-03-09T00:25:44.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:44.301 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788738+0000", 2020-03-09T00:25:44.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044600949999999999 2020-03-09T00:25:44.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.793198+0000", 2020-03-09T00:25:44.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000122469 2020-03-09T00:25:44.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.793321+0000", 2020-03-09T00:25:44.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081692400000000001 2020-03-09T00:25:44.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794138+0000", 2020-03-09T00:25:44.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026755360000000001 2020-03-09T00:25:44.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796813+0000", 2020-03-09T00:25:44.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058884480000000001 2020-03-09T00:25:44.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802702+0000", 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108418 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802810+0000", 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014184085000000001 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.308 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.309 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.309 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:600 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:44.309 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:44.309 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 25.360887456, 2020-03-09T00:25:44.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016073481000000001, 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 600 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.310 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.311 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:44.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.311 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.311 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.311 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3720999999999998e-05 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788783+0000", 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5499999999999999e-06 2020-03-09T00:25:44.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788787+0000", 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0769e-05 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788797+0000", 2020-03-09T00:25:44.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.157e-06 2020-03-09T00:25:44.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788801+0000", 2020-03-09T00:25:44.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1471000000000002e-05 2020-03-09T00:25:44.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788822+0000", 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3599999999999998e-06 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:44.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788826+0000", 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066200779999999997 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.795446+0000", 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013750200000000001 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.795584+0000", 2020-03-09T00:25:44.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00096939200000000002 2020-03-09T00:25:44.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796553+0000", 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001230536 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.797784+0000", 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069359829999999997 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.804720+0000", 2020-03-09T00:25:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010296199999999999 2020-03-09T00:25:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.804823+0000", 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016073481000000001 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:642 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:44.320 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:44.321 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.364282770999999, 2020-03-09T00:25:44.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014044918999999999, 2020-03-09T00:25:44.321 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.321 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.321 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.321 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.321 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 642 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.322 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0393999999999999e-05 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.323 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785404+0000", 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0550000000000005e-06 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785414+0000", 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6269999999999998e-05 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785430+0000", 2020-03-09T00:25:44.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3169999999999998e-06 2020-03-09T00:25:44.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785436+0000", 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00041141700000000002 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785848+0000", 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011097799999999999 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785958+0000", 2020-03-09T00:25:44.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001069469 2020-03-09T00:25:44.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787028+0000", 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082925840000000004 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.795321+0000", 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039837429999999997 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799304+0000", 2020-03-09T00:25:44.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4691999999999994e-05 2020-03-09T00:25:44.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799399+0000", 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014044918999999999 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:643 3.as0 3:51058371:::smithi10428818-28:head [rollback 24] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:44.330 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.364140615, 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014330799999999999, 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.331 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 643 2020-03-09T00:25:44.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.332 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.332 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:44.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6378999999999998e-05 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.333 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785543+0000", 2020-03-09T00:25:44.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4949999999999996e-06 2020-03-09T00:25:44.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.334 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785549+0000", 2020-03-09T00:25:44.334 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4008e-05 2020-03-09T00:25:44.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785563+0000", 2020-03-09T00:25:44.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8459999999999999e-06 2020-03-09T00:25:44.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785568+0000", 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022776580000000001 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787846+0000", 2020-03-09T00:25:44.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011612699999999999 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787962+0000", 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002561486 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.790523+0000", 2020-03-09T00:25:44.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059343010000000003 2020-03-09T00:25:44.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796458+0000", 2020-03-09T00:25:44.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003260359 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799718+0000", 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109141 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799827+0000", 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014330799999999999 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.340 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:644 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:44.341 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:44.341 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 19.364020861, 2020-03-09T00:25:44.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014401074, 2020-03-09T00:25:44.341 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.341 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.341 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.341 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 644 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:44.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.343 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.343 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:44.343 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8556000000000001e-05 2020-03-09T00:25:44.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.343 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785665+0000", 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1480000000000003e-06 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785671+0000", 2020-03-09T00:25:44.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1841999999999999e-05 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785683+0000", 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.907e-06 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.345 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785686+0000", 2020-03-09T00:25:44.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060596110000000003 2020-03-09T00:25:44.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.791746+0000", 2020-03-09T00:25:44.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013271499999999999 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.791879+0000", 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068155099999999997 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.792560+0000", 2020-03-09T00:25:44.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039004560000000001 2020-03-09T00:25:44.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796461+0000", 2020-03-09T00:25:44.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034719059999999999 2020-03-09T00:25:44.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799933+0000", 2020-03-09T00:25:44.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4382000000000006e-05 2020-03-09T00:25:44.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.800017+0000", 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014401074 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 91/48 ECSubWrite(tid=71, reqid=client.4751.0:868, at_version=91'72, trim_to=0'0, roll_forward_to=90'71))", 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:40.854137+0000", 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 3.2954997100000001, 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058700130000000003, 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.351 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.854137+0000", 2020-03-09T00:25:44.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.854137+0000", 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081632600000000003 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.854953+0000", 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.9551999999999998e-05 2020-03-09T00:25:44.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.854973+0000", 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.499e-05 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.854988+0000", 2020-03-09T00:25:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3879999999999997e-06 2020-03-09T00:25:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.854994+0000", 2020-03-09T00:25:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00015309899999999999 2020-03-09T00:25:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.855147+0000", 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3716e-05 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.356 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.855161+0000", 2020-03-09T00:25:44.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048037940000000001 2020-03-09T00:25:44.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.859965+0000", 2020-03-09T00:25:44.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2147999999999997e-05 2020-03-09T00:25:44.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:40.860007+0000", 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058700130000000003 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.359 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:909 3.as0 3:51058371:::smithi10428818-28:head [append 0~458752] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:44.359 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.880547+0000", 2020-03-09T00:25:44.359 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.26908957500000003, 2020-03-09T00:25:44.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018015781000000002, 2020-03-09T00:25:44.359 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 909 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880547+0000", 2020-03-09T00:25:44.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880547+0000", 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021156899999999999 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880759+0000", 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8810000000000001e-06 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880765+0000", 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2879999999999998e-06 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880774+0000", 2020-03-09T00:25:44.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2379999999999999e-06 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880778+0000", 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3570999999999999e-05 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880832+0000", 2020-03-09T00:25:44.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0734000000000002e-05 2020-03-09T00:25:44.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880923+0000", 2020-03-09T00:25:44.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012208919999999999 2020-03-09T00:25:44.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.366 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.366 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882143+0000", 2020-03-09T00:25:44.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046134339999999996 2020-03-09T00:25:44.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.886757+0000", 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011695929000000001 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898453+0000", 2020-03-09T00:25:44.367 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110245 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898563+0000", 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018015781000000002 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.368 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.369 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:910 3.as0 3:51058371:::smithi10428818-28:head [append 0~475136] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:44.369 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.887513+0000", 2020-03-09T00:25:44.369 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 0.26212396900000001, 2020-03-09T00:25:44.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011280438, 2020-03-09T00:25:44.369 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 910 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:44.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887513+0000", 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887513+0000", 2020-03-09T00:25:44.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000323393 2020-03-09T00:25:44.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:44.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887836+0000", 2020-03-09T00:25:44.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.02e-06 2020-03-09T00:25:44.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:44.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887845+0000", 2020-03-09T00:25:44.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4946000000000001e-05 2020-03-09T00:25:44.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:44.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887860+0000", 2020-03-09T00:25:44.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0669999999999997e-06 2020-03-09T00:25:44.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887866+0000", 2020-03-09T00:25:44.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5791999999999998e-05 2020-03-09T00:25:44.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887912+0000", 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000145005 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:44.375 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.888057+0000", 2020-03-09T00:25:44.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012473250000000001 2020-03-09T00:25:44.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.376 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.376 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.376 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.889304+0000", 2020-03-09T00:25:44.376 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088728850000000005 2020-03-09T00:25:44.376 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898177+0000", 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052626699999999997 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:44.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898704+0000", 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9738e-05 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898793+0000", 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011280438 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.378 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:44.379 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:44.379 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:44.381 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:25:44.381 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:44.379+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:44.481 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:44.479+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:44.508 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:44.508 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:44.508 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:44.508 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:44.516 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:25:44.582 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:44.581+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:44.633 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:44.633 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:44.634 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:44.634 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:44.634 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:44.642 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:25:44.683 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:44.681+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:44.760 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:44.760 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:44.760 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:44.760 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:44.760 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.760 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:44.760 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:44.761 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 50.906084591000003, 2020-03-09T00:25:44.761 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999, 2020-03-09T00:25:44.761 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.761 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.761 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.761 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:44.762 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017980700000000001 2020-03-09T00:25:44.763 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.763 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.763 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.763 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851560+0000", 2020-03-09T00:25:44.764 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5363000000000001e-05 2020-03-09T00:25:44.764 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.764 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.765 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.765 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851585+0000", 2020-03-09T00:25:44.765 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4187000000000001e-05 2020-03-09T00:25:44.765 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.765 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.765 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.765 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851609+0000", 2020-03-09T00:25:44.766 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2890000000000003e-06 2020-03-09T00:25:44.766 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.766 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.766 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.766 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851618+0000", 2020-03-09T00:25:44.766 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00036370100000000002 2020-03-09T00:25:44.766 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851982+0000", 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3288999999999999e-05 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.767 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851995+0000", 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074675740000000003 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859463+0000", 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1719999999999999e-05 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.768 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.769 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.769 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859525+0000", 2020-03-09T00:25:44.769 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999 2020-03-09T00:25:44.769 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.769 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.769 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.769 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=56, reqid=client.4751.0:254, at_version=44'23, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 50.893763583999998, 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996, 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.770 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.771 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.771 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.771 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:44.771 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.771 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.771 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.771 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015903600000000001 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863860+0000", 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3975999999999999e-05 2020-03-09T00:25:44.772 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863884+0000", 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6139999999999998e-06 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.773 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863893+0000", 2020-03-09T00:25:44.774 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3190000000000002e-06 2020-03-09T00:25:44.774 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.774 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.774 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.774 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863899+0000", 2020-03-09T00:25:44.774 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039154710000000002 2020-03-09T00:25:44.774 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.775 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.775 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.775 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867814+0000", 2020-03-09T00:25:44.775 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5379999999999998e-05 2020-03-09T00:25:44.775 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.775 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.775 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.776 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867830+0000", 2020-03-09T00:25:44.776 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001627606 2020-03-09T00:25:44.776 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.776 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.776 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869457+0000", 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7565999999999998e-05 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869505+0000", 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996 2020-03-09T00:25:44.777 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.778 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.778 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.778 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.778 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.778 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=77, reqid=client.4751.0:301, at_version=49'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:44.778 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:44.778 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 45.926765760999999, 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001, 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:44.779 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000394366 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.780 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.781 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831093+0000", 2020-03-09T00:25:44.781 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6948999999999999e-05 2020-03-09T00:25:44.781 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.781 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.781 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.781 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831120+0000", 2020-03-09T00:25:44.781 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7549000000000001e-05 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831137+0000", 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1499999999999999e-06 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.782 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831146+0000", 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4503000000000001e-05 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831230+0000", 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4256999999999999e-05 2020-03-09T00:25:44.783 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.784 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.784 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.784 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831244+0000", 2020-03-09T00:25:44.784 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014134180999999999 2020-03-09T00:25:44.784 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.784 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.784 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845379+0000", 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5120000000000002e-05 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845424+0000", 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001 2020-03-09T00:25:44.785 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=78, reqid=client.4751.0:302, at_version=49'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 45.922168608, 2020-03-09T00:25:44.786 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621, 2020-03-09T00:25:44.787 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.787 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.787 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.787 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.787 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.787 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:44.787 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.788 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.788 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:44.788 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166248 2020-03-09T00:25:44.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835462+0000", 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6983000000000001e-05 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835499+0000", 2020-03-09T00:25:44.789 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1426000000000001e-05 2020-03-09T00:25:44.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.790 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.790 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835520+0000", 2020-03-09T00:25:44.790 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1362000000000001e-05 2020-03-09T00:25:44.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.791 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.791 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835532+0000", 2020-03-09T00:25:44.791 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001298925 2020-03-09T00:25:44.791 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.792 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.792 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.792 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836831+0000", 2020-03-09T00:25:44.792 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4944e-05 2020-03-09T00:25:44.792 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.792 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.792 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.793 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836846+0000", 2020-03-09T00:25:44.793 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086294679999999995 2020-03-09T00:25:44.793 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.793 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.793 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.793 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845475+0000", 2020-03-09T00:25:44.793 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7265000000000002e-05 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845522+0000", 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.794 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.795 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.795 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.795 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=79, reqid=client.4751.0:303, at_version=49'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:44.795 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:44.795 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 45.919962920000003, 2020-03-09T00:25:44.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008, 2020-03-09T00:25:44.795 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:44.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0375999999999999e-05 2020-03-09T00:25:44.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837562+0000", 2020-03-09T00:25:44.798 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1155e-05 2020-03-09T00:25:44.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.798 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.798 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.798 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837583+0000", 2020-03-09T00:25:44.798 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.573e-05 2020-03-09T00:25:44.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.799 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837599+0000", 2020-03-09T00:25:44.799 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6429999999999998e-06 2020-03-09T00:25:44.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.800 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837605+0000", 2020-03-09T00:25:44.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074652130000000001 2020-03-09T00:25:44.800 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.800 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.800 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.800 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845071+0000", 2020-03-09T00:25:44.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3423e-05 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845084+0000", 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468407 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.801 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845552+0000", 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6518000000000002e-05 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845619+0000", 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008 2020-03-09T00:25:44.802 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.803 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.803 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.803 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.803 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/48 ECSubWrite(tid=80, reqid=client.4751.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=49'30))", 2020-03-09T00:25:44.803 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:44.803 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 45.918096190999997, 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999, 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:44.804 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7139999999999998e-05 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.806 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839425+0000", 2020-03-09T00:25:44.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6959999999999999e-05 2020-03-09T00:25:44.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.806 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.806 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839452+0000", 2020-03-09T00:25:44.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2141000000000003e-05 2020-03-09T00:25:44.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.807 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.807 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839484+0000", 2020-03-09T00:25:44.807 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2400000000000004e-06 2020-03-09T00:25:44.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.808 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.808 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839491+0000", 2020-03-09T00:25:44.808 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4830000000000002e-05 2020-03-09T00:25:44.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.808 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.808 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.808 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839545+0000", 2020-03-09T00:25:44.809 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3210000000000001e-05 2020-03-09T00:25:44.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.809 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.809 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.809 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839559+0000", 2020-03-09T00:25:44.809 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0054244760000000001 2020-03-09T00:25:44.809 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.844983+0000", 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9245000000000005e-05 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845052+0000", 2020-03-09T00:25:44.811 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999 2020-03-09T00:25:44.811 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.811 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.811 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.813 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 53/52 ECSubWrite(tid=50, reqid=client.4751.0:311, at_version=53'13, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:44.813 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:44.813 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 41.969180076999997, 2020-03-09T00:25:44.813 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444, 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.814 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.815 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.815 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.815 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:44.815 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000284932 2020-03-09T00:25:44.815 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.815 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.815 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788569+0000", 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8404e-05 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788608+0000", 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3697e-05 2020-03-09T00:25:44.816 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788631+0000", 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1178e-05 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.817 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788642+0000", 2020-03-09T00:25:44.818 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000149171 2020-03-09T00:25:44.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.818 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788792+0000", 2020-03-09T00:25:44.818 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7829000000000002e-05 2020-03-09T00:25:44.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788809+0000", 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0065850479999999996 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795395+0000", 2020-03-09T00:25:44.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5185000000000001e-05 2020-03-09T00:25:44.820 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.820 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795450+0000", 2020-03-09T00:25:44.820 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444 2020-03-09T00:25:44.820 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 41.964520358000001, 2020-03-09T00:25:44.821 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999, 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000391498 2020-03-09T00:25:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793335+0000", 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2011999999999998e-05 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793367+0000", 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4936000000000001e-05 2020-03-09T00:25:44.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793382+0000", 2020-03-09T00:25:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0244000000000001e-05 2020-03-09T00:25:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793393+0000", 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011101499999999999 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793504+0000", 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5529e-05 2020-03-09T00:25:44.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793519+0000", 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057666640000000003 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.827 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799286+0000", 2020-03-09T00:25:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011186100000000001 2020-03-09T00:25:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799398+0000", 2020-03-09T00:25:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999 2020-03-09T00:25:44.828 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 41.906682001, 2020-03-09T00:25:44.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001, 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001481797 2020-03-09T00:25:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.831 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852264+0000", 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3690999999999999e-05 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852308+0000", 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.057e-05 2020-03-09T00:25:44.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852338+0000", 2020-03-09T00:25:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0482999999999999e-05 2020-03-09T00:25:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.833 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852349+0000", 2020-03-09T00:25:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5451999999999994e-05 2020-03-09T00:25:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852434+0000", 2020-03-09T00:25:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5016e-05 2020-03-09T00:25:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.835 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852449+0000", 2020-03-09T00:25:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022365531000000001 2020-03-09T00:25:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.836 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874815+0000", 2020-03-09T00:25:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7504999999999999e-05 2020-03-09T00:25:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874872+0000", 2020-03-09T00:25:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001 2020-03-09T00:25:44.837 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=93, reqid=client.4751.0:380, at_version=54'42, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 40.939426771999997, 2020-03-09T00:25:44.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001, 2020-03-09T00:25:44.839 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.839 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.839 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.839 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.839 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:44.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:44.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000153535 2020-03-09T00:25:44.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818191+0000", 2020-03-09T00:25:44.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6421e-05 2020-03-09T00:25:44.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818218+0000", 2020-03-09T00:25:44.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7093000000000001e-05 2020-03-09T00:25:44.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.842 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818235+0000", 2020-03-09T00:25:44.842 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7960000000000001e-06 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818243+0000", 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011362 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.843 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818357+0000", 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2011000000000001e-05 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818369+0000", 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0092246010000000007 2020-03-09T00:25:44.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827594+0000", 2020-03-09T00:25:44.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1653000000000002e-05 2020-03-09T00:25:44.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.846 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.846 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827645+0000", 2020-03-09T00:25:44.846 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001 2020-03-09T00:25:44.847 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.847 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.847 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.847 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.847 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=94, reqid=client.4751.0:381, at_version=54'43, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 40.938132013999997, 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005, 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:44.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:44.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018645000000000001 2020-03-09T00:25:44.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819519+0000", 2020-03-09T00:25:44.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3156e-05 2020-03-09T00:25:44.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819542+0000", 2020-03-09T00:25:44.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5676e-05 2020-03-09T00:25:44.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819558+0000", 2020-03-09T00:25:44.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.742e-06 2020-03-09T00:25:44.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819564+0000", 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027410770000000002 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822305+0000", 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5634999999999999e-05 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822321+0000", 2020-03-09T00:25:44.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0053474569999999999 2020-03-09T00:25:44.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827669+0000", 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4604999999999998e-05 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827703+0000", 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.855 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 54/32 ECSubWrite(tid=105, reqid=client.4751.0:427, at_version=54'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 40.534713459999999, 2020-03-09T00:25:44.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008, 2020-03-09T00:25:44.857 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.857 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.857 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:44.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017923099999999999 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.858 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222930+0000", 2020-03-09T00:25:44.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4340000000000001e-05 2020-03-09T00:25:44.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222954+0000", 2020-03-09T00:25:44.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4165000000000001e-05 2020-03-09T00:25:44.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222969+0000", 2020-03-09T00:25:44.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6719999999999993e-06 2020-03-09T00:25:44.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222977+0000", 2020-03-09T00:25:44.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7257999999999998e-05 2020-03-09T00:25:44.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.862 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223055+0000", 2020-03-09T00:25:44.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4771e-05 2020-03-09T00:25:44.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.864 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223069+0000", 2020-03-09T00:25:44.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.008721477 2020-03-09T00:25:44.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231791+0000", 2020-03-09T00:25:44.866 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3718000000000003e-05 2020-03-09T00:25:44.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.866 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.866 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231845+0000", 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/32 ECSubWrite(tid=162, reqid=osd.11.0:162, at_version=63'46, trim_to=0'0, roll_forward_to=60'44))", 2020-03-09T00:25:44.867 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.980838952999999, 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001, 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:44.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:44.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5686999999999997e-05 2020-03-09T00:25:44.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776681+0000", 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2730000000000001e-05 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776704+0000", 2020-03-09T00:25:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4839e-05 2020-03-09T00:25:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776719+0000", 2020-03-09T00:25:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7049999999999999e-06 2020-03-09T00:25:44.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776726+0000", 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00107576 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777802+0000", 2020-03-09T00:25:44.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2907e-05 2020-03-09T00:25:44.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777815+0000", 2020-03-09T00:25:44.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0051422669999999998 2020-03-09T00:25:44.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782957+0000", 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1214e-05 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.783019+0000", 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 63/32 ECSubWrite(tid=110, reqid=client.4751.0:522, at_version=63'73, trim_to=0'0, roll_forward_to=60'71))", 2020-03-09T00:25:44.875 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:44.876 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.980350969, 2020-03-09T00:25:44.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001, 2020-03-09T00:25:44.876 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.876 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.876 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:44.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7158999999999999e-05 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777171+0000", 2020-03-09T00:25:44.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0669000000000001e-05 2020-03-09T00:25:44.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777191+0000", 2020-03-09T00:25:44.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5477000000000001e-05 2020-03-09T00:25:44.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777207+0000", 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2890000000000003e-06 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777213+0000", 2020-03-09T00:25:44.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010217600000000001 2020-03-09T00:25:44.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777315+0000", 2020-03-09T00:25:44.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2303999999999999e-05 2020-03-09T00:25:44.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777327+0000", 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.005510989 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782838+0000", 2020-03-09T00:25:44.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7664000000000003e-05 2020-03-09T00:25:44.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.883 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.883 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782886+0000", 2020-03-09T00:25:44.883 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001 2020-03-09T00:25:44.883 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.883 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.884 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.884 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.884 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 63/54 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=63'40, trim_to=0'0, roll_forward_to=60'38))", 2020-03-09T00:25:44.884 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:44.885 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.978903731999999, 2020-03-09T00:25:44.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004, 2020-03-09T00:25:44.886 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.886 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.886 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.886 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.886 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6827999999999999e-05 2020-03-09T00:25:44.887 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778617+0000", 2020-03-09T00:25:44.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1481999999999999e-05 2020-03-09T00:25:44.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778639+0000", 2020-03-09T00:25:44.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4163e-05 2020-03-09T00:25:44.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778653+0000", 2020-03-09T00:25:44.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3899999999999998e-06 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778659+0000", 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7515000000000004e-05 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778757+0000", 2020-03-09T00:25:44.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2418e-05 2020-03-09T00:25:44.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778769+0000", 2020-03-09T00:25:44.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058080179999999999 2020-03-09T00:25:44.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784577+0000", 2020-03-09T00:25:44.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3770000000000002e-05 2020-03-09T00:25:44.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784631+0000", 2020-03-09T00:25:44.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004 2020-03-09T00:25:44.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.894 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.895 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 63/54 ECSubWrite(tid=85, reqid=osd.14.0:85, at_version=63'28, trim_to=0'0, roll_forward_to=62'26))", 2020-03-09T00:25:44.895 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:44.895 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 31.977579929000001, 2020-03-09T00:25:44.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001, 2020-03-09T00:25:44.895 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.895 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.896 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9020000000000001e-05 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779943+0000", 2020-03-09T00:25:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8972e-05 2020-03-09T00:25:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779972+0000", 2020-03-09T00:25:44.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5282999999999999e-05 2020-03-09T00:25:44.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779988+0000", 2020-03-09T00:25:44.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5200000000000003e-06 2020-03-09T00:25:44.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779994+0000", 2020-03-09T00:25:44.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027147400000000001 2020-03-09T00:25:44.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780266+0000", 2020-03-09T00:25:44.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1885000000000001e-05 2020-03-09T00:25:44.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780278+0000", 2020-03-09T00:25:44.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0050865659999999998 2020-03-09T00:25:44.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785364+0000", 2020-03-09T00:25:44.903 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6227999999999999e-05 2020-03-09T00:25:44.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.903 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.903 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785410+0000", 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:44.904 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:44.905 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 29.955110985000001, 2020-03-09T00:25:44.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020881999999999, 2020-03-09T00:25:44.905 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.905 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.905 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:44.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:44.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0028740570000000002 2020-03-09T00:25:44.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805227+0000", 2020-03-09T00:25:44.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9300999999999999e-05 2020-03-09T00:25:44.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805257+0000", 2020-03-09T00:25:44.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2918000000000001e-05 2020-03-09T00:25:44.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805270+0000", 2020-03-09T00:25:44.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5249999999999999e-06 2020-03-09T00:25:44.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805274+0000", 2020-03-09T00:25:44.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8279000000000002e-05 2020-03-09T00:25:44.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805332+0000", 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3175e-05 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805346+0000", 2020-03-09T00:25:44.911 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011977272000000001 2020-03-09T00:25:44.912 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.817323+0000", 2020-03-09T00:25:44.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1354999999999999e-05 2020-03-09T00:25:44.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.817374+0000", 2020-03-09T00:25:44.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020881999999999 2020-03-09T00:25:44.913 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.913 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.914 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.914 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 68/67 ECSubWrite(tid=96, reqid=client.4751.0:565, at_version=68'38, trim_to=0'0, roll_forward_to=68'36))", 2020-03-09T00:25:44.914 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:44.914 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.807305474, 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063074259999999997, 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:44.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:44.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2433000000000001e-05 2020-03-09T00:25:44.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950251+0000", 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2212e-05 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950274+0000", 2020-03-09T00:25:44.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.666e-05 2020-03-09T00:25:44.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950290+0000", 2020-03-09T00:25:44.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2519999999999996e-06 2020-03-09T00:25:44.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.919 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950296+0000", 2020-03-09T00:25:44.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8819000000000001e-05 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950345+0000", 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2407e-05 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.920 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950358+0000", 2020-03-09T00:25:44.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060565740000000003 2020-03-09T00:25:44.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.921 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.921 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956414+0000", 2020-03-09T00:25:44.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2068999999999999e-05 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956466+0000", 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063074259999999997 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.922 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.923 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 68/67 ECSubWrite(tid=203, reqid=client.4751.0:575, at_version=68'66, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:44.923 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:44.923 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.804090163000001, 2020-03-09T00:25:44.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007455082, 2020-03-09T00:25:44.923 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:44.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00038396300000000002 2020-03-09T00:25:44.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953758+0000", 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6554e-05 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953785+0000", 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7886000000000001e-05 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953803+0000", 2020-03-09T00:25:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2519999999999996e-06 2020-03-09T00:25:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953809+0000", 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9489e-05 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953858+0000", 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3121e-05 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953871+0000", 2020-03-09T00:25:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068970350000000001 2020-03-09T00:25:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.929 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960768+0000", 2020-03-09T00:25:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0782000000000002e-05 2020-03-09T00:25:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960829+0000", 2020-03-09T00:25:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007455082 2020-03-09T00:25:44.930 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 68/67 ECSubWrite(tid=205, reqid=client.4751.0:577, at_version=68'68, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:44.931 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 26.796738134999998, 2020-03-09T00:25:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0061821239999999998, 2020-03-09T00:25:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023476109999999999 2020-03-09T00:25:44.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963074+0000", 2020-03-09T00:25:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8515000000000001e-05 2020-03-09T00:25:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:44.935 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963092+0000", 2020-03-09T00:25:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0417999999999999e-05 2020-03-09T00:25:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963103+0000", 2020-03-09T00:25:44.936 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8919999999999997e-06 2020-03-09T00:25:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:44.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963109+0000", 2020-03-09T00:25:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9573000000000002e-05 2020-03-09T00:25:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.938 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963148+0000", 2020-03-09T00:25:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.181e-05 2020-03-09T00:25:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963160+0000", 2020-03-09T00:25:44.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037000549999999998 2020-03-09T00:25:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.966860+0000", 2020-03-09T00:25:44.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8250000000000001e-05 2020-03-09T00:25:44.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:44.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:44.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:44.941 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.966908+0000", 2020-03-09T00:25:44.941 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0061821239999999998 2020-03-09T00:25:44.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.942 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.942 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:44.942 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:44.942 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:44.943 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-03-09T00:25:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:367: done (8 left) 2020-03-09T00:25:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:369: done (7 left) 2020-03-09T00:25:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:370: done (6 left) 2020-03-09T00:25:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:371: done (5 left) 2020-03-09T00:25:44.947 INFO:tasks.rados.rados.0.smithi104.stdout:372: done (4 left) 2020-03-09T00:25:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:373: done (3 left) 2020-03-09T00:25:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:374: done (2 left) 2020-03-09T00:25:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:375: done (1 left) 2020-03-09T00:25:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:376: done (0 left) 2020-03-09T00:25:44.948 INFO:tasks.rados.rados.0.smithi104.stdout:377: delete oid 21 current snap is 31 2020-03-09T00:25:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:377: done (0 left) 2020-03-09T00:25:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:378: rmattr oid 8 current snap is 31 2020-03-09T00:25:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:378: done (0 left) 2020-03-09T00:25:44.949 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 36 current snap is 31 2020-03-09T00:25:44.950 INFO:tasks.rados.rados.0.smithi104.stdout:379: seq_num 147 ranges {3047424=65536} 2020-03-09T00:25:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:379: writing smithi10428818-36 from 3047424 to 3112960 tid 1 2020-03-09T00:25:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 4 current snap is 31 2020-03-09T00:25:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:380: seq_num 148 ranges {573440=81920} 2020-03-09T00:25:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:380: writing smithi10428818-4 from 573440 to 655360 tid 1 2020-03-09T00:25:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 40 current snap is 31 2020-03-09T00:25:44.951 INFO:tasks.rados.rados.0.smithi104.stdout:381: seq_num 149 ranges {1753088=98304} 2020-03-09T00:25:44.952 INFO:tasks.rados.rados.0.smithi104.stdout:381: writing smithi10428818-40 from 1753088 to 1851392 tid 1 2020-03-09T00:25:44.952 INFO:tasks.rados.rados.0.smithi104.stdout:382: read oid 9 snap -1 2020-03-09T00:25:44.952 INFO:tasks.rados.rados.0.smithi104.stdout:382: expect deleted 2020-03-09T00:25:44.952 INFO:tasks.rados.rados.0.smithi104.stdout:383: read oid 32 snap -1 2020-03-09T00:25:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:383: expect (ObjNum 31 snap 0 seq_num 31) 2020-03-09T00:25:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:384: snap_create 2020-03-09T00:25:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:379: finishing write tid 1 to smithi10428818-36 2020-03-09T00:25:44.953 INFO:tasks.rados.rados.0.smithi104.stdout:379: finishing write tid 3 to smithi10428818-36 2020-03-09T00:25:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:379: finishing write tid 4 to smithi10428818-36 2020-03-09T00:25:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 36 v 39 (ObjNum 147 snap 31 seq_num 147) dirty exists 2020-03-09T00:25:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:379: left oid 36 (ObjNum 147 snap 31 seq_num 147) 2020-03-09T00:25:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:380: finishing write tid 1 to smithi10428818-4 2020-03-09T00:25:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:380: finishing write tid 3 to smithi10428818-4 2020-03-09T00:25:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:380: finishing write tid 4 to smithi10428818-4 2020-03-09T00:25:44.954 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 4 v 57 (ObjNum 148 snap 31 seq_num 148) dirty exists 2020-03-09T00:25:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:380: left oid 4 (ObjNum 148 snap 31 seq_num 148) 2020-03-09T00:25:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:381: finishing write tid 1 to smithi10428818-40 2020-03-09T00:25:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:381: finishing write tid 3 to smithi10428818-40 2020-03-09T00:25:44.955 INFO:tasks.rados.rados.0.smithi104.stdout:381: finishing write tid 4 to smithi10428818-40 2020-03-09T00:25:44.956 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 40 v 148 (ObjNum 149 snap 31 seq_num 149) dirty exists 2020-03-09T00:25:44.956 INFO:tasks.rados.rados.0.smithi104.stdout:381: left oid 40 (ObjNum 149 snap 31 seq_num 149) 2020-03-09T00:25:44.956 INFO:tasks.rados.rados.0.smithi104.stdout:383: expect (ObjNum 31 snap 0 seq_num 31) 2020-03-09T00:25:44.956 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:44.942+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:44.957 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:25:45.055 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:45.054+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:45.090 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:45.090 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:45.091 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:45.091 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:45.099 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:25:45.156 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:45.155+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:45.225 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:45.225 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:45.225 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:45.232 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:25:45.257 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:45.256+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:45.358 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:200 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~557056] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.520914609999998, 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845, 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 200 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001017256 2020-03-09T00:25:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835741+0000", 2020-03-09T00:25:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0451999999999999e-05 2020-03-09T00:25:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835751+0000", 2020-03-09T00:25:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1429000000000001e-05 2020-03-09T00:25:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835773+0000", 2020-03-09T00:25:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0686000000000001e-05 2020-03-09T00:25:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835783+0000", 2020-03-09T00:25:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7098e-05 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835850+0000", 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100177 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835951+0000", 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014653610000000001 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837416+0000", 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082709510000000003 2020-03-09T00:25:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845687+0000", 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015616149999999999 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847248+0000", 2020-03-09T00:25:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6819999999999998e-05 2020-03-09T00:25:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847345+0000", 2020-03-09T00:25:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845 2020-03-09T00:25:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:201 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.518206307, 2020-03-09T00:25:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784, 2020-03-09T00:25:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 201 2020-03-09T00:25:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081556599999999999 2020-03-09T00:25:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838247+0000", 2020-03-09T00:25:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3170000000000004e-06 2020-03-09T00:25:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838255+0000", 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0073e-05 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838265+0000", 2020-03-09T00:25:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2199999999999997e-06 2020-03-09T00:25:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838271+0000", 2020-03-09T00:25:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003565814 2020-03-09T00:25:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841837+0000", 2020-03-09T00:25:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013320899999999999 2020-03-09T00:25:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841970+0000", 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001092275 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843062+0000", 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003950617 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847013+0000", 2020-03-09T00:25:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067726199999999998 2020-03-09T00:25:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847690+0000", 2020-03-09T00:25:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000361431 2020-03-09T00:25:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.848052+0000", 2020-03-09T00:25:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784 2020-03-09T00:25:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:202 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.517319387999997, 2020-03-09T00:25:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999, 2020-03-09T00:25:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 202 2020-03-09T00:25:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3191e-05 2020-03-09T00:25:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838362+0000", 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2454e-05 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838374+0000", 2020-03-09T00:25:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8269999999999998e-06 2020-03-09T00:25:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838382+0000", 2020-03-09T00:25:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1470000000000003e-06 2020-03-09T00:25:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838386+0000", 2020-03-09T00:25:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073633309999999999 2020-03-09T00:25:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845750+0000", 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103528 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845853+0000", 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059542799999999995 2020-03-09T00:25:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846449+0000", 2020-03-09T00:25:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056665000000000003 2020-03-09T00:25:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847015+0000", 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022036690000000001 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849219+0000", 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7259e-05 2020-03-09T00:25:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849296+0000", 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:203 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.517224409000001, 2020-03-09T00:25:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296, 2020-03-09T00:25:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 203 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4102000000000001e-05 2020-03-09T00:25:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838438+0000", 2020-03-09T00:25:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4809999999999998e-06 2020-03-09T00:25:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838441+0000", 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.691e-06 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838448+0000", 2020-03-09T00:25:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9409999999999999e-06 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838451+0000", 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084379850000000003 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846889+0000", 2020-03-09T00:25:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119473 2020-03-09T00:25:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847008+0000", 2020-03-09T00:25:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029378160000000002 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849946+0000", 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129367 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.850076+0000", 2020-03-09T00:25:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013511040000000001 2020-03-09T00:25:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.851427+0000", 2020-03-09T00:25:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.513927950999999, 2020-03-09T00:25:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999, 2020-03-09T00:25:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8059000000000002e-05 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841758+0000", 2020-03-09T00:25:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3980000000000005e-06 2020-03-09T00:25:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841768+0000", 2020-03-09T00:25:45.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4708e-05 2020-03-09T00:25:45.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841782+0000", 2020-03-09T00:25:45.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1330000000000001e-06 2020-03-09T00:25:45.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841789+0000", 2020-03-09T00:25:45.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037343680000000001 2020-03-09T00:25:45.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845524+0000", 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9321e-05 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845543+0000", 2020-03-09T00:25:45.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999 2020-03-09T00:25:45.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.407 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:45.408 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:45.408 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.513771810999998, 2020-03-09T00:25:45.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002, 2020-03-09T00:25:45.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.408 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:45.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4193e-05 2020-03-09T00:25:45.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841930+0000", 2020-03-09T00:25:45.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0671e-05 2020-03-09T00:25:45.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841941+0000", 2020-03-09T00:25:45.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5038000000000001e-05 2020-03-09T00:25:45.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841956+0000", 2020-03-09T00:25:45.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1e-06 2020-03-09T00:25:45.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841962+0000", 2020-03-09T00:25:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036100849999999999 2020-03-09T00:25:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845572+0000", 2020-03-09T00:25:45.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117707 2020-03-09T00:25:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845690+0000", 2020-03-09T00:25:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002 2020-03-09T00:25:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.513266131000002, 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002, 2020-03-09T00:25:45.415 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.416 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.416 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:45.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:45.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6950000000000002e-05 2020-03-09T00:25:45.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842429+0000", 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1120000000000002e-06 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842438+0000", 2020-03-09T00:25:45.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5486999999999999e-05 2020-03-09T00:25:45.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842453+0000", 2020-03-09T00:25:45.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5080000000000002e-06 2020-03-09T00:25:45.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842460+0000", 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003166791 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845627+0000", 2020-03-09T00:25:45.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7489000000000001e-05 2020-03-09T00:25:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845644+0000", 2020-03-09T00:25:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002 2020-03-09T00:25:45.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.422 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.425 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(osd.12.24:1 3.8s0 3:15387ad6:::smithi10428818-43:head [assert-version v5,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-03-09T00:25:45.425 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:45.426 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.503531084999999, 2020-03-09T00:25:45.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004, 2020-03-09T00:25:45.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.426 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:45.426 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:45.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0395000000000006e-05 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852177+0000", 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1959999999999996e-06 2020-03-09T00:25:45.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852184+0000", 2020-03-09T00:25:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0183000000000001e-05 2020-03-09T00:25:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852194+0000", 2020-03-09T00:25:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.463e-06 2020-03-09T00:25:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852199+0000", 2020-03-09T00:25:45.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7527e-05 2020-03-09T00:25:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852257+0000", 2020-03-09T00:25:45.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132466 2020-03-09T00:25:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852389+0000", 2020-03-09T00:25:45.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042799869999999999 2020-03-09T00:25:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856669+0000", 2020-03-09T00:25:45.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004 2020-03-09T00:25:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:45.434 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.492600998999997, 2020-03-09T00:25:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004, 2020-03-09T00:25:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:45.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022289400000000001 2020-03-09T00:25:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863260+0000", 2020-03-09T00:25:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0071000000000001e-05 2020-03-09T00:25:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863280+0000", 2020-03-09T00:25:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2445e-05 2020-03-09T00:25:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863292+0000", 2020-03-09T00:25:45.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2320000000000001e-06 2020-03-09T00:25:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863298+0000", 2020-03-09T00:25:45.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7059999999999998e-05 2020-03-09T00:25:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863365+0000", 2020-03-09T00:25:45.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6728e-05 2020-03-09T00:25:45.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863381+0000", 2020-03-09T00:25:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048764250000000002 2020-03-09T00:25:45.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868258+0000", 2020-03-09T00:25:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0857999999999999e-05 2020-03-09T00:25:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868319+0000", 2020-03-09T00:25:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004 2020-03-09T00:25:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 48/32 ECSubWrite(tid=43, reqid=osd.14.0:43, at_version=48'12, trim_to=0'0, roll_forward_to=44'10))", 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.583023697999998, 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001, 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.446 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0922999999999999e-05 2020-03-09T00:25:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772675+0000", 2020-03-09T00:25:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8955999999999999e-05 2020-03-09T00:25:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772704+0000", 2020-03-09T00:25:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6671e-05 2020-03-09T00:25:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772721+0000", 2020-03-09T00:25:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5750999999999999e-05 2020-03-09T00:25:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772737+0000", 2020-03-09T00:25:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0823999999999998e-05 2020-03-09T00:25:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772807+0000", 2020-03-09T00:25:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5262000000000001e-05 2020-03-09T00:25:45.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772823+0000", 2020-03-09T00:25:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047643729999999997 2020-03-09T00:25:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777587+0000", 2020-03-09T00:25:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8967999999999993e-05 2020-03-09T00:25:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777666+0000", 2020-03-09T00:25:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001 2020-03-09T00:25:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.455 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:275 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~704512] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.570731965, 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002, 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.456 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 275 2020-03-09T00:25:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:45.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013228269999999999 2020-03-09T00:25:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786229+0000", 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9299999999999997e-06 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786236+0000", 2020-03-09T00:25:45.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3371000000000003e-05 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786269+0000", 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0840000000000003e-06 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786276+0000", 2020-03-09T00:25:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7576e-05 2020-03-09T00:25:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786324+0000", 2020-03-09T00:25:45.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010013699999999999 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786424+0000", 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00139653 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787821+0000", 2020-03-09T00:25:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089582080000000005 2020-03-09T00:25:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796779+0000", 2020-03-09T00:25:45.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036565499999999999 2020-03-09T00:25:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797144+0000", 2020-03-09T00:25:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116684 2020-03-09T00:25:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797261+0000", 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:276 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.568394347999998, 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001, 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.466 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 276 2020-03-09T00:25:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000487605 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787731+0000", 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.875e-06 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787739+0000", 2020-03-09T00:25:45.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8544000000000002e-05 2020-03-09T00:25:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787778+0000", 2020-03-09T00:25:45.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4640999999999999e-05 2020-03-09T00:25:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787792+0000", 2020-03-09T00:25:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038656099999999998 2020-03-09T00:25:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791658+0000", 2020-03-09T00:25:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7368999999999996e-05 2020-03-09T00:25:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791745+0000", 2020-03-09T00:25:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093415700000000002 2020-03-09T00:25:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792680+0000", 2020-03-09T00:25:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004100289 2020-03-09T00:25:45.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796780+0000", 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074439000000000005 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797524+0000", 2020-03-09T00:25:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2327999999999998e-05 2020-03-09T00:25:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797577+0000", 2020-03-09T00:25:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001 2020-03-09T00:25:45.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:277 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:45.477 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.567750867000001, 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006, 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.478 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 277 2020-03-09T00:25:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010529100000000001 2020-03-09T00:25:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787992+0000", 2020-03-09T00:25:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.826e-06 2020-03-09T00:25:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787997+0000", 2020-03-09T00:25:45.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0430000000000008e-06 2020-03-09T00:25:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788005+0000", 2020-03-09T00:25:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1422e-05 2020-03-09T00:25:45.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788017+0000", 2020-03-09T00:25:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073510989999999998 2020-03-09T00:25:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795368+0000", 2020-03-09T00:25:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159238 2020-03-09T00:25:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795527+0000", 2020-03-09T00:25:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073327099999999997 2020-03-09T00:25:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796260+0000", 2020-03-09T00:25:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052061200000000005 2020-03-09T00:25:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796781+0000", 2020-03-09T00:25:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095208999999999995 2020-03-09T00:25:45.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797733+0000", 2020-03-09T00:25:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0259000000000001e-05 2020-03-09T00:25:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797783+0000", 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.488 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:278 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-03-09T00:25:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.56755811, 2020-03-09T00:25:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999, 2020-03-09T00:25:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.489 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 278 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:45.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3933e-05 2020-03-09T00:25:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788134+0000", 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2150000000000007e-06 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788143+0000", 2020-03-09T00:25:45.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.507e-05 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788158+0000", 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.772e-05 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788176+0000", 2020-03-09T00:25:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010090758 2020-03-09T00:25:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798267+0000", 2020-03-09T00:25:45.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012688700000000001 2020-03-09T00:25:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798394+0000", 2020-03-09T00:25:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001284739 2020-03-09T00:25:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.799678+0000", 2020-03-09T00:25:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999 2020-03-09T00:25:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.563584931999998, 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004, 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.497 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677000000000002e-05 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792109+0000", 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1342000000000001e-05 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792120+0000", 2020-03-09T00:25:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8834e-05 2020-03-09T00:25:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792139+0000", 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1979e-05 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792151+0000", 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043798459999999997 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796531+0000", 2020-03-09T00:25:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.609e-05 2020-03-09T00:25:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796547+0000", 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:25:45.503 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.562918627999998, 2020-03-09T00:25:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002, 2020-03-09T00:25:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2404999999999996e-05 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792782+0000", 2020-03-09T00:25:45.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2590000000000006e-06 2020-03-09T00:25:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792791+0000", 2020-03-09T00:25:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6633000000000002e-05 2020-03-09T00:25:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792808+0000", 2020-03-09T00:25:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6499999999999999e-06 2020-03-09T00:25:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792814+0000", 2020-03-09T00:25:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003762077 2020-03-09T00:25:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796576+0000", 2020-03-09T00:25:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8287e-05 2020-03-09T00:25:45.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796595+0000", 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:293 3.8s0 3:129db2f3:::smithi10428818-45:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.539051012000002, 2020-03-09T00:25:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052182940000000001, 2020-03-09T00:25:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.511 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 293 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:45.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5916999999999999e-05 2020-03-09T00:25:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816613+0000", 2020-03-09T00:25:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7960000000000007e-06 2020-03-09T00:25:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816623+0000", 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0010000000000001e-05 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816633+0000", 2020-03-09T00:25:45.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0229999999999998e-06 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816639+0000", 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6136e-05 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816685+0000", 2020-03-09T00:25:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1557000000000005e-05 2020-03-09T00:25:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816766+0000", 2020-03-09T00:25:45.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095985199999999997 2020-03-09T00:25:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817726+0000", 2020-03-09T00:25:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037736010000000001 2020-03-09T00:25:45.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821500+0000", 2020-03-09T00:25:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021410299999999999 2020-03-09T00:25:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821714+0000", 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1298999999999994e-05 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821805+0000", 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052182940000000001 2020-03-09T00:25:45.520 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:294 3.8s0 3:129db2f3:::smithi10428818-45:head [rollback 5] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:45.521 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.538963942000002, 2020-03-09T00:25:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060119980000000002, 2020-03-09T00:25:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.522 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 294 2020-03-09T00:25:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4272000000000001e-05 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816698+0000", 2020-03-09T00:25:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4240000000000002e-06 2020-03-09T00:25:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816702+0000", 2020-03-09T00:25:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8809999999999998e-06 2020-03-09T00:25:45.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816709+0000", 2020-03-09T00:25:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7520000000000001e-06 2020-03-09T00:25:45.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816711+0000", 2020-03-09T00:25:45.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019784109999999998 2020-03-09T00:25:45.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.818690+0000", 2020-03-09T00:25:45.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121984 2020-03-09T00:25:45.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.818812+0000", 2020-03-09T00:25:45.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083844200000000001 2020-03-09T00:25:45.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819650+0000", 2020-03-09T00:25:45.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001850694 2020-03-09T00:25:45.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821501+0000", 2020-03-09T00:25:45.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010669200000000001 2020-03-09T00:25:45.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822568+0000", 2020-03-09T00:25:45.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118218 2020-03-09T00:25:45.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822686+0000", 2020-03-09T00:25:45.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060119980000000002 2020-03-09T00:25:45.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.533 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:295 3.8s0 3:129db2f3:::smithi10428818-45:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:45.533 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:45.533 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.538901533000001, 2020-03-09T00:25:45.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067281850000000002, 2020-03-09T00:25:45.533 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.533 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 295 2020-03-09T00:25:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:45.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5190000000000001e-05 2020-03-09T00:25:45.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816762+0000", 2020-03-09T00:25:45.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.061e-06 2020-03-09T00:25:45.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816765+0000", 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7499999999999997e-06 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816772+0000", 2020-03-09T00:25:45.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.588e-06 2020-03-09T00:25:45.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816774+0000", 2020-03-09T00:25:45.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035648450000000001 2020-03-09T00:25:45.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.820339+0000", 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113943 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:45.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.820453+0000", 2020-03-09T00:25:45.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071432600000000005 2020-03-09T00:25:45.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:45.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821167+0000", 2020-03-09T00:25:45.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068605100000000002 2020-03-09T00:25:45.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821853+0000", 2020-03-09T00:25:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001563734 2020-03-09T00:25:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:45.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823417+0000", 2020-03-09T00:25:45.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7697000000000001e-05 2020-03-09T00:25:45.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823465+0000", 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067281850000000002 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 49/32 ECSubWriteReply(tid=27, last_complete=49'16, committed=1, applied=1))", 2020-03-09T00:25:45.543 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.536615763, 2020-03-09T00:25:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003135231, 2020-03-09T00:25:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0222e-05 2020-03-09T00:25:45.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819053+0000", 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8910000000000003e-06 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:45.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819057+0000", 2020-03-09T00:25:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9330000000000001e-06 2020-03-09T00:25:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819065+0000", 2020-03-09T00:25:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.968e-06 2020-03-09T00:25:45.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:45.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819069+0000", 2020-03-09T00:25:45.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023769609999999999 2020-03-09T00:25:45.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:45.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821446+0000", 2020-03-09T00:25:45.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071125599999999995 2020-03-09T00:25:45.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:45.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:45.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:45.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822158+0000", 2020-03-09T00:25:45.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003135231 2020-03-09T00:25:45.550 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.550 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.550 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.550 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:45.550 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:45.551 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:45.552 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:45.356+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:45.552 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:45.550+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:45.553 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:25:45.653 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:45.651+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:45.669 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:45.669 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:45.669 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:45.669 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:45.677 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:25:45.754 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:45.752+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:45.808 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-03-09T00:25:45.837 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:45.837 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:45.837 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:45.837 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:45.837 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:45.845 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:25:45.872 INFO:tasks.rados.rados.0.smithi104.stdout:379: done (5 left) 2020-03-09T00:25:45.872 INFO:tasks.rados.rados.0.smithi104.stdout:380: done (4 left) 2020-03-09T00:25:45.872 INFO:tasks.rados.rados.0.smithi104.stdout:381: done (3 left) 2020-03-09T00:25:45.872 INFO:tasks.rados.rados.0.smithi104.stdout:382: done (2 left) 2020-03-09T00:25:45.873 INFO:tasks.rados.rados.0.smithi104.stdout:383: done (1 left) 2020-03-09T00:25:45.873 INFO:tasks.rados.rados.0.smithi104.stdout:384: done (0 left) 2020-03-09T00:25:45.873 INFO:tasks.rados.rados.0.smithi104.stdout:385: read oid 6 snap -1 2020-03-09T00:25:45.873 INFO:tasks.rados.rados.0.smithi104.stdout:385: expect deleted 2020-03-09T00:25:45.873 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 49 current snap is 32 2020-03-09T00:25:45.873 INFO:tasks.rados.rados.0.smithi104.stdout:386: seq_num 150 ranges {720896=114688} 2020-03-09T00:25:45.874 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is unset 2020-03-09T00:25:45.875 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:45.873+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:45.877 INFO:tasks.rados.rados.0.smithi104.stdout:386: writing smithi10428818-49 from 720896 to 835584 tid 1 2020-03-09T00:25:45.877 INFO:tasks.rados.rados.0.smithi104.stdout:387: read oid 22 snap -1 2020-03-09T00:25:45.877 INFO:tasks.rados.rados.0.smithi104.stdout:387: expect (ObjNum 124 snap 24 seq_num 124) 2020-03-09T00:25:45.877 INFO:tasks.rados.rados.0.smithi104.stdout:388: rmattr oid 39 current snap is 32 2020-03-09T00:25:45.878 INFO:tasks.rados.rados.0.smithi104.stdout:388: done (3 left) 2020-03-09T00:25:45.878 INFO:tasks.rados.rados.0.smithi104.stdout:389: read oid 3 snap 27 2020-03-09T00:25:45.878 INFO:tasks.rados.rados.0.smithi104.stdout:389: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:45.878 INFO:tasks.rados.rados.0.smithi104.stdout:390: rollback oid 34 current snap is 32 2020-03-09T00:25:45.878 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 34 to 29 2020-03-09T00:25:45.878 INFO:tasks.rados.rados.0.smithi104.stdout:385: done (4 left) 2020-03-09T00:25:45.878 INFO:tasks.rados.rados.0.smithi104.stdout:391: snap_remove snap 30 2020-03-09T00:25:45.956 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:45.954+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:45.959 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:45.980 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:46.002 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:46.011 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:46.011 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:46.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:46.012 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:46.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.012 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s3 53/52 ECSubWrite(tid=50, reqid=client.4751.0:311, at_version=53'13, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:46.012 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:46.012 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 43.220640154999998, 2020-03-09T00:25:46.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013212144, 2020-03-09T00:25:46.013 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.013 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.013 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.013 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.013 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.013 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:46.013 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:46.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00046852800000000001 2020-03-09T00:25:46.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788727+0000", 2020-03-09T00:25:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6373999999999999e-05 2020-03-09T00:25:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788754+0000", 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5746000000000001e-05 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788769+0000", 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4629999999999998e-06 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788776+0000", 2020-03-09T00:25:46.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019937899999999999 2020-03-09T00:25:46.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788975+0000", 2020-03-09T00:25:46.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4996e-05 2020-03-09T00:25:46.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788990+0000", 2020-03-09T00:25:46.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012423663999999999 2020-03-09T00:25:46.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.801414+0000", 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6993999999999997e-05 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.801471+0000", 2020-03-09T00:25:46.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013212144 2020-03-09T00:25:46.020 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.020 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.020 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.020 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:46.021 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:46.021 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 43.157688122000003, 2020-03-09T00:25:46.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023044938000000001, 2020-03-09T00:25:46.021 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.021 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.021 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:46.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:46.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059576639999999997 2020-03-09T00:25:46.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.023 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.023 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857168+0000", 2020-03-09T00:25:46.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7962000000000001e-05 2020-03-09T00:25:46.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857196+0000", 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6986999999999998e-05 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.024 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857213+0000", 2020-03-09T00:25:46.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4509999999999997e-06 2020-03-09T00:25:46.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857220+0000", 2020-03-09T00:25:46.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000106183 2020-03-09T00:25:46.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857326+0000", 2020-03-09T00:25:46.026 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.182e-05 2020-03-09T00:25:46.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857338+0000", 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016865054000000001 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.874203+0000", 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2816999999999999e-05 2020-03-09T00:25:46.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.874256+0000", 2020-03-09T00:25:46.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023044938000000001 2020-03-09T00:25:46.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.028 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.028 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.2s3 54/48 ECSubWrite(tid=7, reqid=client.4751.0:375, at_version=54'16, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.204065419999999, 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014845443, 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.029 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.030 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:46.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:46.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040868809999999997 2020-03-09T00:25:46.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.031 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.031 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808920+0000", 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1741e-05 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808952+0000", 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5818e-05 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808968+0000", 2020-03-09T00:25:46.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9209999999999998e-06 2020-03-09T00:25:46.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808977+0000", 2020-03-09T00:25:46.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018277500000000001 2020-03-09T00:25:46.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.809160+0000", 2020-03-09T00:25:46.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6409999999999997e-06 2020-03-09T00:25:46.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.809169+0000", 2020-03-09T00:25:46.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010439004999999999 2020-03-09T00:25:46.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.819608+0000", 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0661000000000007e-05 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.819679+0000", 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014845443 2020-03-09T00:25:46.036 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.037 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.037 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.037 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:485 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:46.037 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:46.038 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 37.196272964000002, 2020-03-09T00:25:46.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028623089000000001, 2020-03-09T00:25:46.038 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.038 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.039 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.039 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.039 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.041 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 485 2020-03-09T00:25:46.041 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.041 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.041 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.041 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.041 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:46.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.042 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.042 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.042 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:46.042 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0157000000000001e-05 2020-03-09T00:25:46.042 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812676+0000", 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9910000000000008e-06 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.043 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812685+0000", 2020-03-09T00:25:46.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6279000000000001e-05 2020-03-09T00:25:46.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812701+0000", 2020-03-09T00:25:46.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7309999999999993e-06 2020-03-09T00:25:46.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812709+0000", 2020-03-09T00:25:46.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016644500000000001 2020-03-09T00:25:46.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812875+0000", 2020-03-09T00:25:46.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014119600000000001 2020-03-09T00:25:46.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813017+0000", 2020-03-09T00:25:46.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013810770000000001 2020-03-09T00:25:46.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.814398+0000", 2020-03-09T00:25:46.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015392095 2020-03-09T00:25:46.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829790+0000", 2020-03-09T00:25:46.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011370328000000001 2020-03-09T00:25:46.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841160+0000", 2020-03-09T00:25:46.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8789999999999995e-05 2020-03-09T00:25:46.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841249+0000", 2020-03-09T00:25:46.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028623089000000001 2020-03-09T00:25:46.051 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.051 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.051 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.052 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:486 3.8s0 3:1d2a75f4:::smithi10428818-31:head [rollback 14] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:46.052 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:46.052 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 37.196129681000002, 2020-03-09T00:25:46.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028685548000000002, 2020-03-09T00:25:46.052 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.052 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.052 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 486 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:46.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:46.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4728999999999998e-05 2020-03-09T00:25:46.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812814+0000", 2020-03-09T00:25:46.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1049999999999997e-06 2020-03-09T00:25:46.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812821+0000", 2020-03-09T00:25:46.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2251e-05 2020-03-09T00:25:46.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.057 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.057 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812833+0000", 2020-03-09T00:25:46.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.754e-06 2020-03-09T00:25:46.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812840+0000", 2020-03-09T00:25:46.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046243279999999996 2020-03-09T00:25:46.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817464+0000", 2020-03-09T00:25:46.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121283 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817586+0000", 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00083277700000000004 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818418+0000", 2020-03-09T00:25:46.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011373098999999999 2020-03-09T00:25:46.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829791+0000", 2020-03-09T00:25:46.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011588105 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841380+0000", 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5117000000000006e-05 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841455+0000", 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028685548000000002 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.063 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:487 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 37.195870020999998, 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048834447000000003, 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.064 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.065 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 487 2020-03-09T00:25:46.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.065 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:46.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2626999999999997e-05 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813071+0000", 2020-03-09T00:25:46.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3329999999999999e-06 2020-03-09T00:25:46.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.067 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.067 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813079+0000", 2020-03-09T00:25:46.067 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5886999999999999e-05 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813095+0000", 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.601e-06 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.068 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813101+0000", 2020-03-09T00:25:46.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020620191999999999 2020-03-09T00:25:46.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833721+0000", 2020-03-09T00:25:46.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013959399999999999 2020-03-09T00:25:46.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833861+0000", 2020-03-09T00:25:46.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018544080000000001 2020-03-09T00:25:46.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.835715+0000", 2020-03-09T00:25:46.071 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014671090000000001 2020-03-09T00:25:46.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.071 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.071 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.071 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.837183+0000", 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024587595 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861770+0000", 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3101000000000002e-05 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.072 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861863+0000", 2020-03-09T00:25:46.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048834447000000003 2020-03-09T00:25:46.073 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.073 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.073 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 59/54 ECSubWriteReply(tid=5, last_complete=59'26, committed=1, applied=1))", 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 37.192991651, 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013886617, 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.074 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:46.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4713999999999998e-05 2020-03-09T00:25:46.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815962+0000", 2020-03-09T00:25:46.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6719999999999993e-06 2020-03-09T00:25:46.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815971+0000", 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5335000000000001e-05 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815986+0000", 2020-03-09T00:25:46.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4869999999999999e-06 2020-03-09T00:25:46.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.078 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.078 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815992+0000", 2020-03-09T00:25:46.078 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013780812 2020-03-09T00:25:46.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829773+0000", 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0596999999999999e-05 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.079 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829794+0000", 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013886617 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:46.080 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 37.165420933999997, 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017489998, 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:46.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:46.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001188824 2020-03-09T00:25:46.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844667+0000", 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5516000000000001e-05 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844692+0000", 2020-03-09T00:25:46.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4897e-05 2020-03-09T00:25:46.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844707+0000", 2020-03-09T00:25:46.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3720000000000001e-06 2020-03-09T00:25:46.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844713+0000", 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012514399999999999 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845965+0000", 2020-03-09T00:25:46.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3451999999999999e-05 2020-03-09T00:25:46.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.086 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.086 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845978+0000", 2020-03-09T00:25:46.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014935277 2020-03-09T00:25:46.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860914+0000", 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4219999999999999e-05 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.087 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860968+0000", 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017489998 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:528 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:46.088 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:46.089 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 32.237444959000001, 2020-03-09T00:25:46.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015882618000000001, 2020-03-09T00:25:46.089 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.089 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.089 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.089 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.089 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.090 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 528 2020-03-09T00:25:46.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.090 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:46.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:46.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5457e-05 2020-03-09T00:25:46.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.092 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771479+0000", 2020-03-09T00:25:46.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8210000000000001e-06 2020-03-09T00:25:46.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771484+0000", 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2479999999999996e-06 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771492+0000", 2020-03-09T00:25:46.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8449999999999999e-06 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771495+0000", 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4236999999999998e-05 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771529+0000", 2020-03-09T00:25:46.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5760000000000006e-06 2020-03-09T00:25:46.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:46.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771538+0000", 2020-03-09T00:25:46.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078539520000000008 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.779392+0000", 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011161400000000001 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.779504+0000", 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00052493799999999997 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.780029+0000", 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.006329693 2020-03-09T00:25:46.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.098 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.786358+0000", 2020-03-09T00:25:46.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00088499900000000003 2020-03-09T00:25:46.098 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.098 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.098 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.787243+0000", 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3238000000000004e-05 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.787336+0000", 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015882618000000001 2020-03-09T00:25:46.099 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.cs4 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 31.206489804, 2020-03-09T00:25:46.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014065318, 2020-03-09T00:25:46.101 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.101 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.101 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:46.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002574813 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.804984+0000", 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9797e-05 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805004+0000", 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1419999999999999e-06 2020-03-09T00:25:46.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805013+0000", 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2899999999999996e-06 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805017+0000", 2020-03-09T00:25:46.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2918999999999998e-05 2020-03-09T00:25:46.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.105 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805080+0000", 2020-03-09T00:25:46.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7710000000000001e-06 2020-03-09T00:25:46.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805090+0000", 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011347336 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.816437+0000", 2020-03-09T00:25:46.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7249999999999997e-05 2020-03-09T00:25:46.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.816474+0000", 2020-03-09T00:25:46.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014065318 2020-03-09T00:25:46.107 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:676 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~2523136,checksum] snapc 0=[] ondisk+read+known_if_redirected e80)", 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.194241028, 2020-03-09T00:25:46.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014479270000000001, 2020-03-09T00:25:46.109 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.109 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:46.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.109 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 676 2020-03-09T00:25:46.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.110 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:46.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6242000000000005e-05 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814744+0000", 2020-03-09T00:25:46.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1100000000000003e-06 2020-03-09T00:25:46.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814752+0000", 2020-03-09T00:25:46.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3115999999999999e-05 2020-03-09T00:25:46.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814765+0000", 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5550000000000002e-06 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814771+0000", 2020-03-09T00:25:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043361559999999999 2020-03-09T00:25:46.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.819107+0000", 2020-03-09T00:25:46.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013518100000000001 2020-03-09T00:25:46.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.819242+0000", 2020-03-09T00:25:46.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098949099999999998 2020-03-09T00:25:46.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829137+0000", 2020-03-09T00:25:46.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014479270000000001 2020-03-09T00:25:46.116 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.116 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.116 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.116 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:677 3.8s0 3.2fae54b8 (undecoded) ondisk+read+known_if_redirected e80)", 2020-03-09T00:25:46.117 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:46.117 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.194064311999998, 2020-03-09T00:25:46.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014509384, 2020-03-09T00:25:46.117 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.117 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.117 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.118 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.118 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 677 2020-03-09T00:25:46.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.118 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.118 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.118 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:46.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.119 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.119 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.119 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:46.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6549000000000002e-05 2020-03-09T00:25:46.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814911+0000", 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3329999999999997e-06 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814919+0000", 2020-03-09T00:25:46.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2802999999999999e-05 2020-03-09T00:25:46.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814932+0000", 2020-03-09T00:25:46.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3369999999999996e-06 2020-03-09T00:25:46.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814939+0000", 2020-03-09T00:25:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014396528 2020-03-09T00:25:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829335+0000", 2020-03-09T00:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8340000000000005e-06 2020-03-09T00:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829344+0000", 2020-03-09T00:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014509384 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:749 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~475136] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.866101+0000", 2020-03-09T00:25:46.124 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.142797646, 2020-03-09T00:25:46.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01484748, 2020-03-09T00:25:46.125 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.125 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.125 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.125 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.125 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.125 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 749 2020-03-09T00:25:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866101+0000", 2020-03-09T00:25:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866101+0000", 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00061777499999999999 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866719+0000", 2020-03-09T00:25:46.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1809999999999998e-06 2020-03-09T00:25:46.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866725+0000", 2020-03-09T00:25:46.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6569999999999992e-06 2020-03-09T00:25:46.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866735+0000", 2020-03-09T00:25:46.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0209999999999996e-06 2020-03-09T00:25:46.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866741+0000", 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002096213 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.868837+0000", 2020-03-09T00:25:46.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00023417300000000001 2020-03-09T00:25:46.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869071+0000", 2020-03-09T00:25:46.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011264479999999999 2020-03-09T00:25:46.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.870198+0000", 2020-03-09T00:25:46.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010087149 2020-03-09T00:25:46.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880285+0000", 2020-03-09T00:25:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00037859299999999999 2020-03-09T00:25:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880663+0000", 2020-03-09T00:25:46.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00028527000000000001 2020-03-09T00:25:46.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880949+0000", 2020-03-09T00:25:46.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01484748 2020-03-09T00:25:46.135 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.136 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.136 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.136 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:750 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~507904] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:46.136 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.868989+0000", 2020-03-09T00:25:46.136 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.139910124, 2020-03-09T00:25:46.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015288784, 2020-03-09T00:25:46.137 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.137 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.138 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.138 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.138 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.138 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 750 2020-03-09T00:25:46.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.139 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.140 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.140 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.868989+0000", 2020-03-09T00:25:46.140 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.868989+0000", 2020-03-09T00:25:46.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00096076899999999999 2020-03-09T00:25:46.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869949+0000", 2020-03-09T00:25:46.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0250000000000002e-06 2020-03-09T00:25:46.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869957+0000", 2020-03-09T00:25:46.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4758e-05 2020-03-09T00:25:46.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869972+0000", 2020-03-09T00:25:46.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.618e-06 2020-03-09T00:25:46.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869979+0000", 2020-03-09T00:25:46.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027946799999999999 2020-03-09T00:25:46.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872774+0000", 2020-03-09T00:25:46.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000105646 2020-03-09T00:25:46.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872879+0000", 2020-03-09T00:25:46.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00082986099999999999 2020-03-09T00:25:46.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.146 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.873709+0000", 2020-03-09T00:25:46.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066888529999999998 2020-03-09T00:25:46.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.146 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.147 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880398+0000", 2020-03-09T00:25:46.147 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035930810000000001 2020-03-09T00:25:46.147 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.147 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.147 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883991+0000", 2020-03-09T00:25:46.147 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00028649300000000002 2020-03-09T00:25:46.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.148 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.148 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.148 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884277+0000", 2020-03-09T00:25:46.148 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015288784 2020-03-09T00:25:46.149 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.149 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.150 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.150 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:752 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~98304] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:46.151 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.872378+0000", 2020-03-09T00:25:46.151 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.136521057, 2020-03-09T00:25:46.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013385007000000001, 2020-03-09T00:25:46.151 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.151 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.151 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.151 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.152 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.152 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 752 2020-03-09T00:25:46.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.152 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872378+0000", 2020-03-09T00:25:46.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.153 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.153 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872378+0000", 2020-03-09T00:25:46.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015559499999999999 2020-03-09T00:25:46.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872533+0000", 2020-03-09T00:25:46.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8180000000000004e-06 2020-03-09T00:25:46.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872541+0000", 2020-03-09T00:25:46.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5211000000000001e-05 2020-03-09T00:25:46.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.155 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872556+0000", 2020-03-09T00:25:46.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5539999999999997e-06 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872562+0000", 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087003029999999995 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.881262+0000", 2020-03-09T00:25:46.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013599200000000001 2020-03-09T00:25:46.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.881398+0000", 2020-03-09T00:25:46.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00081354799999999998 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.882212+0000", 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021470320000000001 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.158 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884359+0000", 2020-03-09T00:25:46.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001083751 2020-03-09T00:25:46.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.159 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.885443+0000", 2020-03-09T00:25:46.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00032020300000000002 2020-03-09T00:25:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.885763+0000", 2020-03-09T00:25:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013385007000000001 2020-03-09T00:25:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.160 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.161 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.161 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:753 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (57) in=7b] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:46.161 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.872624+0000", 2020-03-09T00:25:46.161 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.136274811, 2020-03-09T00:25:46.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013590579, 2020-03-09T00:25:46.162 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.162 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.162 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.162 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.162 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.162 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 753 2020-03-09T00:25:46.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872624+0000", 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.164 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872624+0000", 2020-03-09T00:25:46.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0097000000000002e-05 2020-03-09T00:25:46.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.164 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.164 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872704+0000", 2020-03-09T00:25:46.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1079999999999993e-06 2020-03-09T00:25:46.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872712+0000", 2020-03-09T00:25:46.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2544000000000001e-05 2020-03-09T00:25:46.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872725+0000", 2020-03-09T00:25:46.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4110000000000004e-06 2020-03-09T00:25:46.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872730+0000", 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010322237999999999 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883052+0000", 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5949e-05 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.168 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.168 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883138+0000", 2020-03-09T00:25:46.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050229199999999999 2020-03-09T00:25:46.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883641+0000", 2020-03-09T00:25:46.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001221462 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884862+0000", 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010869440000000001 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.170 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.885949+0000", 2020-03-09T00:25:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000265534 2020-03-09T00:25:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886215+0000", 2020-03-09T00:25:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013590579 2020-03-09T00:25:46.172 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.172 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.172 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.172 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:754 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e85)", 2020-03-09T00:25:46.172 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.872789+0000", 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.136109663999999, 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016005664999999999, 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.173 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 754 2020-03-09T00:25:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872789+0000", 2020-03-09T00:25:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872789+0000", 2020-03-09T00:25:46.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6080000000000001e-05 2020-03-09T00:25:46.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872815+0000", 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1710000000000001e-06 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872820+0000", 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7790000000000001e-06 2020-03-09T00:25:46.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872829+0000", 2020-03-09T00:25:46.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4390000000000002e-06 2020-03-09T00:25:46.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872834+0000", 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013904133000000001 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886738+0000", 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000145012 2020-03-09T00:25:46.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886883+0000", 2020-03-09T00:25:46.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019120509999999999 2020-03-09T00:25:46.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.888795+0000", 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016005664999999999 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.180 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:46.181 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.884865+0000", 2020-03-09T00:25:46.181 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 11.124033973, 2020-03-09T00:25:46.181 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015800109, 2020-03-09T00:25:46.181 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.181 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:46.181 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884865+0000", 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.182 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884865+0000", 2020-03-09T00:25:46.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020353350000000001 2020-03-09T00:25:46.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886900+0000", 2020-03-09T00:25:46.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6072999999999999e-05 2020-03-09T00:25:46.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886926+0000", 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6269999999999998e-05 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886943+0000", 2020-03-09T00:25:46.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2869999999999999e-06 2020-03-09T00:25:46.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.185 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886950+0000", 2020-03-09T00:25:46.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5357e-05 2020-03-09T00:25:46.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886995+0000", 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2044e-05 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.186 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887007+0000", 2020-03-09T00:25:46.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013517240999999999 2020-03-09T00:25:46.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.900524+0000", 2020-03-09T00:25:46.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000140502 2020-03-09T00:25:46.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.900665+0000", 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015800109 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.189 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:772 3.8s0 3:1d2a75f4:::smithi10428818-31:head [copy-from ver 24] snapc 32=[32] ondisk+write+known_if_redirected e86)", 2020-03-09T00:25:46.189 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:35.830970+0000", 2020-03-09T00:25:46.189 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.177928689, 2020-03-09T00:25:46.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037538067000000001, 2020-03-09T00:25:46.189 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.189 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 772 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.190 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.830970+0000", 2020-03-09T00:25:46.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.830970+0000", 2020-03-09T00:25:46.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1035000000000002e-05 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831021+0000", 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5910000000000003e-06 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.192 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831027+0000", 2020-03-09T00:25:46.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0087e-05 2020-03-09T00:25:46.193 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.193 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831037+0000", 2020-03-09T00:25:46.193 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1220000000000001e-06 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831046+0000", 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000196751 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831243+0000", 2020-03-09T00:25:46.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0415999999999996e-05 2020-03-09T00:25:46.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831333+0000", 2020-03-09T00:25:46.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018291963000000001 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.849625+0000", 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018442281000000001 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:46.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.868067+0000", 2020-03-09T00:25:46.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000137429 2020-03-09T00:25:46.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.197 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:46.197 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.868205+0000", 2020-03-09T00:25:46.197 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030339199999999998 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.868508+0000", 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037538067000000001 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.198 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.199 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:773 3.8s0 3:1d2a75f4:::smithi10428818-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e86)", 2020-03-09T00:25:46.199 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:35.831093+0000", 2020-03-09T00:25:46.199 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 10.177805814999999, 2020-03-09T00:25:46.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039651458000000001, 2020-03-09T00:25:46.199 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 773 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:46.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:46.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831093+0000", 2020-03-09T00:25:46.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:46.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:46.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831093+0000", 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5473000000000003e-05 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831138+0000", 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5350000000000001e-06 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:46.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831145+0000", 2020-03-09T00:25:46.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2931e-05 2020-03-09T00:25:46.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:46.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831158+0000", 2020-03-09T00:25:46.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6210000000000001e-06 2020-03-09T00:25:46.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:46.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831164+0000", 2020-03-09T00:25:46.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00031384300000000001 2020-03-09T00:25:46.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831477+0000", 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4112e-05 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:46.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831532+0000", 2020-03-09T00:25:46.206 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036296544 2020-03-09T00:25:46.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.206 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.206 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.206 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.867828+0000", 2020-03-09T00:25:46.206 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9529000000000003e-05 2020-03-09T00:25:46.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.867918+0000", 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025560940000000001 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:46.207 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.870474+0000", 2020-03-09T00:25:46.208 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0533999999999998e-05 2020-03-09T00:25:46.208 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.208 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.208 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:46.208 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.870554+0000", 2020-03-09T00:25:46.208 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000190242 2020-03-09T00:25:46.208 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.870744+0000", 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039651458000000001 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.209 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:46.210 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:46.210 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:46.212 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:25:46.212 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:46.213 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:46.213 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:46.213 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:46.213 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:46.213 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:46.213 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:46.214 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:46.209+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:46.312 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:46.311+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:46.335 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:46.335 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:46.344 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:25:46.366 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:46.386 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:46.386 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:46.385+0000 7f1277fff700 0 -- 172.21.15.148:0/1384101775 >> [v2:172.21.15.50:6816/27631,v1:172.21.15.50:6817/27631] conn(0x7f125c0297b0 msgr2=0x7f125c02bba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-03-09T00:25:46.413 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:46.411+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:46.459 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:46.459 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:46.467 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:25:46.513 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:46.512+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:46.583 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:46.583 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:46.591 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:46.608 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:46.629 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:46.636 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:46.635+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:46.648 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:46.650 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.650 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.650 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.650 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.651 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.652 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.652 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.652 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.652 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.652 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'false' 2020-03-09T00:25:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:46.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:46.716 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:46.715+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:46.728 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:46.728 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:46.737 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:46.817 INFO:tasks.ceph.osd.8.smithi148.stderr:2020-03-09T00:25:46.816+0000 7efc35f83700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 27357) UID: 0 2020-03-09T00:25:46.881 INFO:tasks.rados.rados.0.smithi104.stdout:391: done (4 left) 2020-03-09T00:25:46.881 INFO:tasks.rados.rados.0.smithi104.stdout:392: snap_create 2020-03-09T00:25:46.882 INFO:tasks.rados.rados.0.smithi104.stdout:386: finishing write tid 1 to smithi10428818-49 2020-03-09T00:25:46.882 INFO:tasks.rados.rados.0.smithi104.stdout:386: finishing write tid 3 to smithi10428818-49 2020-03-09T00:25:46.882 INFO:tasks.rados.rados.0.smithi104.stdout:389: expect (ObjNum 36 snap 0 seq_num 36) 2020-03-09T00:25:46.896 INFO:tasks.rados.rados.0.smithi104.stdout:386: finishing write tid 4 to smithi10428818-49 2020-03-09T00:25:46.897 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 49 v 137 (ObjNum 150 snap 32 seq_num 150) dirty exists 2020-03-09T00:25:46.897 INFO:tasks.rados.rados.0.smithi104.stdout:386: left oid 49 (ObjNum 150 snap 32 seq_num 150) 2020-03-09T00:25:46.897 INFO:tasks.rados.rados.0.smithi104.stdout:387: expect (ObjNum 124 snap 24 seq_num 124) 2020-03-09T00:25:46.902 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:46.903 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:46.911 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:46.955 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:46.954+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:47.019 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:47.017+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:47.036 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.037 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.047 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:25:47.052 INFO:tasks.rados.rados.0.smithi104.stdout:390: finishing rollback tid 1 to smithi10428818-34 2020-03-09T00:25:47.052 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 34 v 164 (ObjNum -326469984 snap 22099 seq_num 1912250256) dirty dne 2020-03-09T00:25:47.120 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:47.119+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:47.165 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.165 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.173 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:25:47.221 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:47.219+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:47.289 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.289 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.298 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:25:47.322 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:47.320+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:47.413 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.414 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.422 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2020-03-09T00:25:47.423 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:47.421+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:47.538 INFO:tasks.ceph.osd.10.smithi139.stderr:2020-03-09T00:25:47.525+0000 7f86725bc700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26974) UID: 0 2020-03-09T00:25:47.538 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.538 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.547 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2020-03-09T00:25:47.627 INFO:tasks.ceph.osd.0.smithi148.stderr:2020-03-09T00:25:47.626+0000 7f7b5a979700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27356) UID: 0 2020-03-09T00:25:47.661 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.662 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.670 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2020-03-09T00:25:47.729 INFO:tasks.ceph.osd.14.smithi139.stderr:2020-03-09T00:25:47.728+0000 7f8b362f5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26976) UID: 0 2020-03-09T00:25:47.785 INFO:teuthology.orchestra.run.smithi104.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.785 INFO:teuthology.orchestra.run.smithi104.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.794 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2020-03-09T00:25:47.830 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:47.828+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:47.884 INFO:tasks.rados.rados.0.smithi104.stdout:386: done (4 left) 2020-03-09T00:25:47.885 INFO:tasks.rados.rados.0.smithi104.stdout:387: done (3 left) 2020-03-09T00:25:47.885 INFO:tasks.rados.rados.0.smithi104.stdout:389: done (2 left) 2020-03-09T00:25:47.885 INFO:tasks.rados.rados.0.smithi104.stdout:390: done (1 left) 2020-03-09T00:25:47.885 INFO:tasks.rados.rados.0.smithi104.stdout:392: done (0 left) 2020-03-09T00:25:47.885 INFO:tasks.rados.rados.0.smithi104.stdout:393: rmattr oid 22 current snap is 33 2020-03-09T00:25:47.885 INFO:tasks.rados.rados.0.smithi104.stdout:393: done (0 left) 2020-03-09T00:25:47.886 INFO:tasks.rados.rados.0.smithi104.stdout:394: rollback oid 33 current snap is 33 2020-03-09T00:25:47.886 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 33 to 27 2020-03-09T00:25:47.886 INFO:tasks.rados.rados.0.smithi104.stdout:395: delete oid 8 current snap is 33 2020-03-09T00:25:47.888 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-03-09T00:25:47.889 INFO:tasks.rados.rados.0.smithi104.stdout:394: finishing rollback tid 1 to smithi10428818-33 2020-03-09T00:25:47.889 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 33 v 124 (ObjNum 0 snap 0 seq_num 0) dirty dne 2020-03-09T00:25:47.890 INFO:tasks.rados.rados.0.smithi104.stdout:394: done (1 left) 2020-03-09T00:25:47.890 INFO:tasks.rados.rados.0.smithi104.stdout:395: done (0 left) 2020-03-09T00:25:47.890 INFO:tasks.rados.rados.0.smithi104.stdout:396: rollback oid 14 current snap is 33 2020-03-09T00:25:47.890 INFO:tasks.rados.rados.0.smithi104.stdout:rollback oid 14 to 32 2020-03-09T00:25:47.890 INFO:tasks.rados.rados.0.smithi104.stdout:397: snap_create 2020-03-09T00:25:47.896 INFO:tasks.rados.rados.0.smithi104.stdout:396: finishing rollback tid 0 to smithi10428818-14 2020-03-09T00:25:47.896 INFO:tasks.rados.rados.0.smithi104.stdout:396: finishing rollback tid 1 to smithi10428818-14 2020-03-09T00:25:47.896 INFO:tasks.rados.rados.0.smithi104.stdout:396: finishing rollback tid 2 to smithi10428818-14 2020-03-09T00:25:47.896 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 14 v 96 (ObjNum 119 snap 24 seq_num 119) dirty exists 2020-03-09T00:25:47.926 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:47.926 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:47.930 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:47.929+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:47.935 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2020-03-09T00:25:48.031 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:48.029+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:48.050 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:48.050 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:48.058 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2020-03-09T00:25:48.132 INFO:tasks.ceph.osd.13.smithi050.stderr:2020-03-09T00:25:48.130+0000 7f1c5c7a3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 27625) UID: 0 2020-03-09T00:25:48.174 INFO:teuthology.orchestra.run.smithi139.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:48.174 INFO:teuthology.orchestra.run.smithi139.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:48.182 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2020-03-09T00:25:48.233 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:48.231+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:48.320 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:48.320 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:48.330 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2020-03-09T00:25:48.333 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:48.332+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:48.434 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:48.433+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:48.494 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:48.494 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:48.503 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2020-03-09T00:25:48.535 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:48.533+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:48.635 INFO:tasks.ceph.osd.15.smithi104.stderr:2020-03-09T00:25:48.634+0000 7f25586ff700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 28010) UID: 0 2020-03-09T00:25:48.661 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-03-09T00:25:48.668 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:48.668 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:48.676 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2020-03-09T00:25:48.749 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:48.748+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:48.800 INFO:teuthology.orchestra.run.smithi148.stdout:osd.0: {} 2020-03-09T00:25:48.820 INFO:teuthology.orchestra.run.smithi148.stdout:osd.1: {} 2020-03-09T00:25:48.827 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:48.828 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:48.836 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2020-03-09T00:25:48.839 INFO:tasks.ceph.osd.2.smithi139.stderr:2020-03-09T00:25:48.838+0000 7f8af930b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26969) UID: 0 2020-03-09T00:25:48.842 INFO:teuthology.orchestra.run.smithi148.stdout:osd.2: {} 2020-03-09T00:25:48.871 INFO:teuthology.orchestra.run.smithi148.stdout:osd.3: {} 2020-03-09T00:25:48.886 INFO:tasks.rados.rados.0.smithi104.stdout:396: done (1 left) 2020-03-09T00:25:48.886 INFO:tasks.rados.rados.0.smithi104.stdout:397: done (0 left) 2020-03-09T00:25:48.886 INFO:tasks.rados.rados.0.smithi104.stdout:append oid 24 current snap is 34 2020-03-09T00:25:48.886 INFO:tasks.rados.rados.0.smithi104.stdout:398: seq_num 152 ranges {0=147456} 2020-03-09T00:25:48.887 INFO:tasks.rados.rados.0.smithi104.stdout:398: writing smithi10428818-24 from 0 to 147456 tid 1 2020-03-09T00:25:48.887 INFO:tasks.rados.rados.0.smithi104.stdout:399: delete oid 9 current snap is 34 2020-03-09T00:25:48.889 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is unset 2020-03-09T00:25:48.893 INFO:tasks.rados.rados.0.smithi104.stdout:398: finishing write tid 1 to smithi10428818-24 2020-03-09T00:25:48.894 INFO:tasks.rados.rados.0.smithi104.stdout:398: finishing write tid 3 to smithi10428818-24 2020-03-09T00:25:48.894 INFO:tasks.rados.rados.0.smithi104.stdout:399: done (1 left) 2020-03-09T00:25:48.895 INFO:tasks.rados.rados.0.smithi104.stdout: waiting on 1 2020-03-09T00:25:48.897 INFO:tasks.rados.rados.0.smithi104.stdout:398: finishing write tid 4 to smithi10428818-24 2020-03-09T00:25:48.898 INFO:tasks.rados.rados.0.smithi104.stdout:update_object_version oid 24 v 98 (ObjNum 152 snap 34 seq_num 152) dirty exists 2020-03-09T00:25:48.898 INFO:tasks.rados.rados.0.smithi104.stdout:398: left oid 24 (ObjNum 152 snap 34 seq_num 152) 2020-03-09T00:25:48.898 INFO:tasks.rados.rados.0.smithi104.stdout:398: done (0 left) 2020-03-09T00:25:48.898 INFO:teuthology.orchestra.run.smithi148.stdout:osd.4: {} 2020-03-09T00:25:48.900 INFO:tasks.rados.rados.0.smithi104.stderr:0 errors. 2020-03-09T00:25:48.900 INFO:tasks.rados.rados.0.smithi104.stderr: 2020-03-09T00:25:48.905 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:25:48.921 INFO:teuthology.orchestra.run.smithi148.stdout:osd.5: {} 2020-03-09T00:25:48.943 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:48.942+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:48.944 INFO:teuthology.orchestra.run.smithi148.stdout:osd.7: {} 2020-03-09T00:25:48.978 INFO:teuthology.orchestra.run.smithi148.stdout:osd.8: {} 2020-03-09T00:25:48.987 INFO:teuthology.orchestra.run.smithi148.stdout:ERROR: (22) Invalid argument 2020-03-09T00:25:48.988 INFO:teuthology.orchestra.run.smithi148.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2020-03-09T00:25:48.997 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2020-03-09T00:25:48.998 INFO:teuthology.orchestra.run.smithi148.stdout:osd.9: {} 2020-03-09T00:25:49.019 INFO:teuthology.orchestra.run.smithi148.stdout:osd.10: {} 2020-03-09T00:25:49.040 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:49.038+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:49.041 INFO:teuthology.orchestra.run.smithi148.stdout:osd.11: {} 2020-03-09T00:25:49.065 INFO:teuthology.orchestra.run.smithi148.stdout:osd.12: {} 2020-03-09T00:25:49.086 INFO:teuthology.orchestra.run.smithi148.stdout:osd.13: {} 2020-03-09T00:25:49.105 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:25:49.104+0000 7f372b7fe700 0 -- 172.21.15.148:0/2767496225 >> [v2:172.21.15.139:6818/26981,v1:172.21.15.139:6821/26981] conn(0x7f370c02c7e0 msgr2=0x7f370c02ebd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2020-03-09T00:25:49.109 INFO:teuthology.orchestra.run.smithi148.stdout:osd.14: {} 2020-03-09T00:25:49.121 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:49.121 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:49.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:49.122 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:49.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.122 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:251 3.0s0 3:0efbb6fa:::smithi10428818-44:head [copy-from ver 5] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:49.122 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:49.122 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.269506116999999, 2020-03-09T00:25:49.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001, 2020-03-09T00:25:49.122 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 251 2020-03-09T00:25:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.123 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848985+0000", 2020-03-09T00:25:49.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6965999999999995e-05 2020-03-09T00:25:49.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849062+0000", 2020-03-09T00:25:49.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6689999999999995e-06 2020-03-09T00:25:49.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849071+0000", 2020-03-09T00:25:49.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7941999999999999e-05 2020-03-09T00:25:49.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849089+0000", 2020-03-09T00:25:49.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9480000000000003e-06 2020-03-09T00:25:49.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849099+0000", 2020-03-09T00:25:49.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4438000000000003e-05 2020-03-09T00:25:49.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849153+0000", 2020-03-09T00:25:49.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012488000000000001 2020-03-09T00:25:49.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.129 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.129 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849278+0000", 2020-03-09T00:25:49.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013412429999999999 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862691+0000", 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011733578 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.130 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.874424+0000", 2020-03-09T00:25:49.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000863506 2020-03-09T00:25:49.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875288+0000", 2020-03-09T00:25:49.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9488999999999998e-05 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.875357+0000", 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026371846000000001 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.132 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.133 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:252 3.0s0 3:0efbb6fa:::smithi10428818-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:49.133 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:49.133 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 55.269357319000001, 2020-03-09T00:25:49.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368, 2020-03-09T00:25:49.133 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.134 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.134 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.134 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.134 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.134 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 252 2020-03-09T00:25:49.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.134 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:49.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.140 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849134+0000", 2020-03-09T00:25:49.140 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7280999999999998e-05 2020-03-09T00:25:49.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849171+0000", 2020-03-09T00:25:49.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6030000000000001e-06 2020-03-09T00:25:49.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849178+0000", 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.308e-05 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849191+0000", 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.553e-06 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849197+0000", 2020-03-09T00:25:49.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030484499999999999 2020-03-09T00:25:49.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849501+0000", 2020-03-09T00:25:49.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5923e-05 2020-03-09T00:25:49.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:49.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849557+0000", 2020-03-09T00:25:49.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019572935999999999 2020-03-09T00:25:49.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869130+0000", 2020-03-09T00:25:49.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013702000000000001 2020-03-09T00:25:49.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869267+0000", 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0068816229999999999 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876149+0000", 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8204999999999999e-05 2020-03-09T00:25:49.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.147 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.147 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876247+0000", 2020-03-09T00:25:49.147 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109299 2020-03-09T00:25:49.147 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.147 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.147 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.876356+0000", 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027222368 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.148 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs1 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:49.149 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:49.149 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 46.267401673000002, 2020-03-09T00:25:49.149 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026627181999999999, 2020-03-09T00:25:49.149 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.149 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:49.149 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:49.150 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851090+0000", 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058834270000000001 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.856973+0000", 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1265999999999999e-05 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.856994+0000", 2020-03-09T00:25:49.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5070000000000006e-06 2020-03-09T00:25:49.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.153 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.153 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857004+0000", 2020-03-09T00:25:49.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0560000000000004e-06 2020-03-09T00:25:49.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857009+0000", 2020-03-09T00:25:49.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1946e-05 2020-03-09T00:25:49.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.155 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857061+0000", 2020-03-09T00:25:49.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4282e-05 2020-03-09T00:25:49.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857075+0000", 2020-03-09T00:25:49.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020582362999999999 2020-03-09T00:25:49.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.877658+0000", 2020-03-09T00:25:49.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9335e-05 2020-03-09T00:25:49.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.157 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.157 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.877717+0000", 2020-03-09T00:25:49.158 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026627181999999999 2020-03-09T00:25:49.158 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.158 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.158 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.159 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:412 3.0s0 3:054f30c0:::smithi10428818-49:head [copy-from ver 0] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:49.159 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:49.159 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.274980112999998, 2020-03-09T00:25:49.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.97896074700000002, 2020-03-09T00:25:49.159 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.159 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.159 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.160 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.160 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.160 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 412 2020-03-09T00:25:49.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.161 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:49.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843511+0000", 2020-03-09T00:25:49.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014666000000000001 2020-03-09T00:25:49.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843658+0000", 2020-03-09T00:25:49.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0059e-05 2020-03-09T00:25:49.163 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843668+0000", 2020-03-09T00:25:49.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8175000000000001e-05 2020-03-09T00:25:49.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.164 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.164 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843686+0000", 2020-03-09T00:25:49.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0209999999999999e-06 2020-03-09T00:25:49.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843694+0000", 2020-03-09T00:25:49.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8402000000000001e-05 2020-03-09T00:25:49.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843753+0000", 2020-03-09T00:25:49.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9160000000000001e-06 2020-03-09T00:25:49.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:49.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:49.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.95898846199999999 2020-03-09T00:25:49.167 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.802744+0000", 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019997600000000001 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.802944+0000", 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019528075999999998 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.169 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.169 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.822472+0000", 2020-03-09T00:25:49.169 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.97896074700000002 2020-03-09T00:25:49.169 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.169 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.169 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.170 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:413 3.0s0 3:054f30c0:::smithi10428818-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:49.170 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:49.170 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.274736582999999, 2020-03-09T00:25:49.170 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.98098931099999997, 2020-03-09T00:25:49.170 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.170 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.171 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.171 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.171 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.171 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 413 2020-03-09T00:25:49.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.171 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:49.172 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.172 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.172 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843755+0000", 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4046999999999999e-05 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843809+0000", 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1140000000000006e-06 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843817+0000", 2020-03-09T00:25:49.174 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5950000000000001e-05 2020-03-09T00:25:49.174 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.174 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.174 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843833+0000", 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4799999999999998e-06 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843839+0000", 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6656000000000003e-05 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.176 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843876+0000", 2020-03-09T00:25:49.176 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8109999999999999e-06 2020-03-09T00:25:49.176 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.176 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.176 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:49.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.843878+0000", 2020-03-09T00:25:49.177 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.95925583699999994 2020-03-09T00:25:49.177 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.177 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.177 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.177 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803134+0000", 2020-03-09T00:25:49.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1947999999999997e-05 2020-03-09T00:25:49.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.178 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.178 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:49.178 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803196+0000", 2020-03-09T00:25:49.178 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020717148000000001 2020-03-09T00:25:49.178 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.179 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.179 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.823913+0000", 2020-03-09T00:25:49.179 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012539800000000001 2020-03-09T00:25:49.179 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.179 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.824038+0000", 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00070592199999999999 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.824744+0000", 2020-03-09T00:25:49.180 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.98098931099999997 2020-03-09T00:25:49.181 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.181 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.181 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.181 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.181 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.181 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:420 3.0s0 3:07071236:::smithi10428818-3:head [rollback 10] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:49.181 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:49.182 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 44.913256195000002, 2020-03-09T00:25:49.182 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61566089000000002, 2020-03-09T00:25:49.182 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.182 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.182 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.182 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.182 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.183 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 420 2020-03-09T00:25:49.183 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.183 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.183 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.183 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.183 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:49.183 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.184 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.184 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205235+0000", 2020-03-09T00:25:49.184 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5570000000000001e-05 2020-03-09T00:25:49.184 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.184 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.185 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.185 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205261+0000", 2020-03-09T00:25:49.185 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8359999999999997e-06 2020-03-09T00:25:49.185 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.186 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.186 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205267+0000", 2020-03-09T00:25:49.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0376e-05 2020-03-09T00:25:49.187 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.187 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.187 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.187 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205277+0000", 2020-03-09T00:25:49.187 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4229999999999999e-06 2020-03-09T00:25:49.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.188 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.188 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205284+0000", 2020-03-09T00:25:49.188 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2729000000000003e-05 2020-03-09T00:25:49.188 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.188 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.189 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205337+0000", 2020-03-09T00:25:49.189 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8240000000000001e-06 2020-03-09T00:25:49.189 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.189 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.189 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:49.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205341+0000", 2020-03-09T00:25:49.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.59787851299999994 2020-03-09T00:25:49.191 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.191 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.191 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.191 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803219+0000", 2020-03-09T00:25:49.191 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000158166 2020-03-09T00:25:49.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.192 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.193 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.193 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.803378+0000", 2020-03-09T00:25:49.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012380760000000001 2020-03-09T00:25:49.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.194 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.194 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.194 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804616+0000", 2020-03-09T00:25:49.194 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0037069310000000001 2020-03-09T00:25:49.194 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.195 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.195 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.808323+0000", 2020-03-09T00:25:49.195 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012483437999999999 2020-03-09T00:25:49.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820806+0000", 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0007999999999996e-05 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820896+0000", 2020-03-09T00:25:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61566089000000002 2020-03-09T00:25:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:421 3.0s0 3:07071236:::smithi10428818-3:head [append 0~0] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 44.913158717000002, 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61577959800000004, 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 421 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205333+0000", 2020-03-09T00:25:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1856000000000001e-05 2020-03-09T00:25:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205364+0000", 2020-03-09T00:25:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.9029999999999997e-06 2020-03-09T00:25:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205368+0000", 2020-03-09T00:25:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2529999999999998e-06 2020-03-09T00:25:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205376+0000", 2020-03-09T00:25:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.004e-06 2020-03-09T00:25:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205379+0000", 2020-03-09T00:25:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3745999999999998e-05 2020-03-09T00:25:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205412+0000", 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1799999999999999e-06 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.205414+0000", 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.59956388100000002 2020-03-09T00:25:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804977+0000", 2020-03-09T00:25:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115327 2020-03-09T00:25:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805093+0000", 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00070765400000000005 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805800+0000", 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025236199999999999 2020-03-09T00:25:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.808324+0000", 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012705651 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.821030+0000", 2020-03-09T00:25:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2522999999999996e-05 2020-03-09T00:25:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.821112+0000", 2020-03-09T00:25:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.61577959800000004 2020-03-09T00:25:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.209 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.209 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.209 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.209 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.209 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:488 3.9s0 3:93e8184b:::smithi10428818-7:head [copy-from ver 2] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:49.209 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.306751660000003, 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033520886, 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.210 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 488 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.211 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811740+0000", 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7763999999999996e-05 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811817+0000", 2020-03-09T00:25:49.212 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.093e-05 2020-03-09T00:25:49.213 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.213 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.213 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811828+0000", 2020-03-09T00:25:49.213 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6980999999999999e-05 2020-03-09T00:25:49.213 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.213 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811845+0000", 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9039999999999998e-06 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811854+0000", 2020-03-09T00:25:49.214 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079268800000000005 2020-03-09T00:25:49.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.215 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.215 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.215 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812647+0000", 2020-03-09T00:25:49.215 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.132e-05 2020-03-09T00:25:49.215 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.215 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812658+0000", 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043188640000000004 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.816977+0000", 2020-03-09T00:25:49.216 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000109483 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817087+0000", 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011389424 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.217 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.218 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.828476+0000", 2020-03-09T00:25:49.218 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048806689999999998 2020-03-09T00:25:49.218 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.218 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.218 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.218 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833357+0000", 2020-03-09T00:25:49.218 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011728186999999999 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845085+0000", 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017567200000000001 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.219 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.220 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845261+0000", 2020-03-09T00:25:49.220 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033520886 2020-03-09T00:25:49.220 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.220 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.220 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.220 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.220 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.221 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:489 3.9s0 3:93e8184b:::smithi10428818-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:49.221 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:49.221 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.306582888000001, 2020-03-09T00:25:49.221 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037452473999999999, 2020-03-09T00:25:49.221 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.221 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.221 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.222 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.222 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.222 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 489 2020-03-09T00:25:49.222 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.222 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.222 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.222 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.223 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:49.223 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.223 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.223 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.223 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.223 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811908+0000", 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2522999999999999e-05 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811951+0000", 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9319999999999999e-06 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.224 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.225 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.225 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811958+0000", 2020-03-09T00:25:49.225 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3498e-05 2020-03-09T00:25:49.225 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.225 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.225 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.225 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811971+0000", 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7749999999999998e-06 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.811977+0000", 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00075333400000000003 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.226 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.227 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.227 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812731+0000", 2020-03-09T00:25:49.227 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.183e-06 2020-03-09T00:25:49.227 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.227 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.227 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:49.227 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812740+0000", 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005894598 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818634+0000", 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4839000000000006e-05 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818709+0000", 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010967121999999999 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829676+0000", 2020-03-09T00:25:49.229 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013027000000000001 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829807+0000", 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019337825999999999 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.230 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.231 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849144+0000", 2020-03-09T00:25:49.231 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012293699999999999 2020-03-09T00:25:49.231 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.231 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.231 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.231 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849267+0000", 2020-03-09T00:25:49.231 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3636999999999998e-05 2020-03-09T00:25:49.232 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.232 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.232 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.232 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849361+0000", 2020-03-09T00:25:49.232 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037452473999999999 2020-03-09T00:25:49.232 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.233 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.233 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.233 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.233 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:490 3.0s0 3:07071236:::smithi10428818-3:head [append 0~786432] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:49.233 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:49.234 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.299525770999999, 2020-03-09T00:25:49.234 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049768433000000001, 2020-03-09T00:25:49.234 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.234 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.235 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.235 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.235 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.235 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 490 2020-03-09T00:25:49.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.235 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.236 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.236 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:49.236 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.236 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.237 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818966+0000", 2020-03-09T00:25:49.237 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040424099999999998 2020-03-09T00:25:49.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.237 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.237 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.237 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823008+0000", 2020-03-09T00:25:49.237 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8820000000000007e-06 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823017+0000", 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6317e-05 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.238 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.239 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823033+0000", 2020-03-09T00:25:49.239 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.622e-06 2020-03-09T00:25:49.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.239 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.239 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823041+0000", 2020-03-09T00:25:49.239 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0034770550000000002 2020-03-09T00:25:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.826518+0000", 2020-03-09T00:25:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0074775010000000001 2020-03-09T00:25:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.241 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.241 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833995+0000", 2020-03-09T00:25:49.241 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017771289999999999 2020-03-09T00:25:49.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.241 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.242 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.835772+0000", 2020-03-09T00:25:49.242 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0086191449999999999 2020-03-09T00:25:49.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.242 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.244 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.244 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844392+0000", 2020-03-09T00:25:49.244 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024243164000000001 2020-03-09T00:25:49.244 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.244 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.244 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.244 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.868635+0000", 2020-03-09T00:25:49.245 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.9208000000000003e-05 2020-03-09T00:25:49.245 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.245 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.245 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.245 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.868734+0000", 2020-03-09T00:25:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.049768433000000001 2020-03-09T00:25:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:491 3.0s0 3:07071236:::smithi10428818-3:head [append 0~491520] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.295390611000002, 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046647411, 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.247 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.248 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 491 2020-03-09T00:25:49.248 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.248 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.248 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.248 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.248 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:49.248 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823101+0000", 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00087253199999999997 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.249 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823973+0000", 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8790000000000003e-06 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823981+0000", 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5770999999999999e-05 2020-03-09T00:25:49.250 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.251 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.251 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.251 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.823997+0000", 2020-03-09T00:25:49.251 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9669999999999998e-06 2020-03-09T00:25:49.251 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.251 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.251 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824003+0000", 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021062780999999999 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845066+0000", 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8070000000000002e-05 2020-03-09T00:25:49.252 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845154+0000", 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00088644099999999997 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.253 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.846040+0000", 2020-03-09T00:25:49.254 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0084721699999999994 2020-03-09T00:25:49.254 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.254 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.254 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.254 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854512+0000", 2020-03-09T00:25:49.254 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015183247 2020-03-09T00:25:49.254 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.255 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.255 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.255 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869696+0000", 2020-03-09T00:25:49.255 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2553000000000003e-05 2020-03-09T00:25:49.255 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.256 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.256 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.256 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869748+0000", 2020-03-09T00:25:49.256 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.046647411 2020-03-09T00:25:49.256 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.257 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.257 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.257 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.257 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.257 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:492 3.0s0 3:07071236:::smithi10428818-3:head [append 0~278528] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:49.258 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:49.258 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.294435911999997, 2020-03-09T00:25:49.258 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045823801999999997, 2020-03-09T00:25:49.258 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.258 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.258 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.258 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.259 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.259 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 492 2020-03-09T00:25:49.259 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.259 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.259 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.259 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.259 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:49.260 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.260 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.260 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.260 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.260 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824055+0000", 2020-03-09T00:25:49.261 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044278900000000002 2020-03-09T00:25:49.261 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.261 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.261 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.261 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824498+0000", 2020-03-09T00:25:49.261 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7820000000000005e-06 2020-03-09T00:25:49.261 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824508+0000", 2020-03-09T00:25:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3557000000000001e-05 2020-03-09T00:25:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.263 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824522+0000", 2020-03-09T00:25:49.263 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7799999999999997e-06 2020-03-09T00:25:49.263 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.263 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.263 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.263 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824527+0000", 2020-03-09T00:25:49.263 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024667254 2020-03-09T00:25:49.264 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.264 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.264 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.264 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849195+0000", 2020-03-09T00:25:49.264 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000133366 2020-03-09T00:25:49.264 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.265 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.265 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.265 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849328+0000", 2020-03-09T00:25:49.265 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001009271 2020-03-09T00:25:49.265 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.265 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.266 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.266 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.850337+0000", 2020-03-09T00:25:49.266 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016518857000000001 2020-03-09T00:25:49.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.266 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.866856+0000", 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002975601 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869832+0000", 2020-03-09T00:25:49.267 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7545e-05 2020-03-09T00:25:49.268 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.268 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.268 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.268 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.869879+0000", 2020-03-09T00:25:49.268 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045823801999999997 2020-03-09T00:25:49.268 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.268 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.269 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.269 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.269 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.269 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:493 3.0s0 3:07071236:::smithi10428818-3:head [setxattr _header (56) in=7b] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:49.269 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:49.269 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.293914168000001, 2020-03-09T00:25:49.269 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045571699, 2020-03-09T00:25:49.270 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.270 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.270 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.270 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.270 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.270 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 493 2020-03-09T00:25:49.270 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.271 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.271 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.271 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.271 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:49.271 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824577+0000", 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1864000000000001e-05 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.272 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824659+0000", 2020-03-09T00:25:49.273 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1690000000000003e-06 2020-03-09T00:25:49.273 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.273 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.273 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.273 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824666+0000", 2020-03-09T00:25:49.273 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3011e-05 2020-03-09T00:25:49.273 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.274 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.274 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.274 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824679+0000", 2020-03-09T00:25:49.274 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6269999999999997e-06 2020-03-09T00:25:49.274 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.274 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.274 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.275 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824685+0000", 2020-03-09T00:25:49.275 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.029865282 2020-03-09T00:25:49.275 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.275 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.275 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.275 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854550+0000", 2020-03-09T00:25:49.275 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015000699999999999 2020-03-09T00:25:49.276 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.276 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.276 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.276 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854700+0000", 2020-03-09T00:25:49.276 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032691450000000002 2020-03-09T00:25:49.276 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.276 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.277 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.277 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.857969+0000", 2020-03-09T00:25:49.277 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0088895759999999997 2020-03-09T00:25:49.277 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.277 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.277 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.277 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.866859+0000", 2020-03-09T00:25:49.278 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032396439999999999 2020-03-09T00:25:49.278 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.278 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.278 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.278 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870099+0000", 2020-03-09T00:25:49.279 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0374000000000003e-05 2020-03-09T00:25:49.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.279 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.279 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.279 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870149+0000", 2020-03-09T00:25:49.279 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.045571699 2020-03-09T00:25:49.280 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.280 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.280 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.280 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.280 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:494 3.0s0 3:07071236:::smithi10428818-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:49.281 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:49.281 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.293763427000002, 2020-03-09T00:25:49.281 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048769923, 2020-03-09T00:25:49.281 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.281 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.281 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.281 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.282 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.282 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 494 2020-03-09T00:25:49.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.282 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.282 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.282 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.282 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:49.283 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.283 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.283 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.283 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824728+0000", 2020-03-09T00:25:49.283 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1047999999999997e-05 2020-03-09T00:25:49.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.284 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.284 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.284 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824769+0000", 2020-03-09T00:25:49.284 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5699999999999998e-06 2020-03-09T00:25:49.284 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.284 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.284 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.285 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824776+0000", 2020-03-09T00:25:49.285 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2532e-05 2020-03-09T00:25:49.285 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.285 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.285 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.285 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824788+0000", 2020-03-09T00:25:49.285 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0990000000000001e-06 2020-03-09T00:25:49.286 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.286 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.286 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.286 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824792+0000", 2020-03-09T00:25:49.286 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035212312000000003 2020-03-09T00:25:49.288 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.288 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.288 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.288 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860004+0000", 2020-03-09T00:25:49.288 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000129932 2020-03-09T00:25:49.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.289 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.289 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:49.289 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860134+0000", 2020-03-09T00:25:49.289 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010193544000000001 2020-03-09T00:25:49.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.289 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.290 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.290 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870328+0000", 2020-03-09T00:25:49.290 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3980999999999995e-05 2020-03-09T00:25:49.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.290 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.290 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.290 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.870402+0000", 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030959049999999999 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.873498+0000", 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048769923 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.291 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.292 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.292 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.292 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.292 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:506 3.0s0 3:05d2883d:::smithi10428818-38:head [stat,delete] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:49.292 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:49.292 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.284277746999997, 2020-03-09T00:25:49.293 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037224634, 2020-03-09T00:25:49.293 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.293 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.293 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.293 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.293 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.294 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 506 2020-03-09T00:25:49.294 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.294 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.294 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.294 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.294 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:49.294 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834214+0000", 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8146999999999998e-05 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.295 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.296 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834262+0000", 2020-03-09T00:25:49.296 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4239999999999993e-06 2020-03-09T00:25:49.296 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.296 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.296 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.296 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834270+0000", 2020-03-09T00:25:49.296 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5318000000000001e-05 2020-03-09T00:25:49.297 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.297 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.297 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.297 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834285+0000", 2020-03-09T00:25:49.299 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5679999999999996e-06 2020-03-09T00:25:49.299 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834292+0000", 2020-03-09T00:25:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.025873846999999998 2020-03-09T00:25:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860166+0000", 2020-03-09T00:25:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017915800000000001 2020-03-09T00:25:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860345+0000", 2020-03-09T00:25:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040042910000000001 2020-03-09T00:25:49.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.302 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.302 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.302 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.864349+0000", 2020-03-09T00:25:49.302 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0029197120000000001 2020-03-09T00:25:49.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.303 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.303 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.867269+0000", 2020-03-09T00:25:49.303 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0041192809999999998 2020-03-09T00:25:49.303 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.304 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.304 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.871388+0000", 2020-03-09T00:25:49.304 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9888000000000001e-05 2020-03-09T00:25:49.304 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.304 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.304 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.871438+0000", 2020-03-09T00:25:49.305 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037224634 2020-03-09T00:25:49.305 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.305 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.305 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.305 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.305 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:535 3.9s0 3:93e8184b:::smithi10428818-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e64)", 2020-03-09T00:25:49.306 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:49.306 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 35.334754680000003, 2020-03-09T00:25:49.306 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026643350999999999, 2020-03-09T00:25:49.306 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.306 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.306 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.306 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.307 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.307 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 535 2020-03-09T00:25:49.307 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.307 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.307 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.307 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.307 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:49.308 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.308 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.308 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.308 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.308 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783737+0000", 2020-03-09T00:25:49.308 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5562999999999997e-05 2020-03-09T00:25:49.308 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.309 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.309 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.309 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783782+0000", 2020-03-09T00:25:49.309 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5560000000000004e-06 2020-03-09T00:25:49.309 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.309 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.309 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.310 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783789+0000", 2020-03-09T00:25:49.310 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2455e-05 2020-03-09T00:25:49.310 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.310 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.310 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.310 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783801+0000", 2020-03-09T00:25:49.311 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.386e-06 2020-03-09T00:25:49.311 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.311 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.311 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.311 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.783806+0000", 2020-03-09T00:25:49.311 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018586522000000001 2020-03-09T00:25:49.311 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.312 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.312 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.312 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.802392+0000", 2020-03-09T00:25:49.312 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7225999999999995e-05 2020-03-09T00:25:49.312 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.312 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.312 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:49.313 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.802479+0000", 2020-03-09T00:25:49.313 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051016660000000004 2020-03-09T00:25:49.313 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.313 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.313 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.313 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.807581+0000", 2020-03-09T00:25:49.313 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6997999999999998e-05 2020-03-09T00:25:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.807648+0000", 2020-03-09T00:25:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027319789999999998 2020-03-09T00:25:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.314 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.315 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.315 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.810380+0000", 2020-03-09T00:25:49.315 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.026643350999999999 2020-03-09T00:25:49.316 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.317 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.317 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.317 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.317 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.317 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:687 3.9s0 3:91ab8712:::smithi10428818-16:head [copy-from ver 60] snapc 2c=[2c,2b,2a,23] ondisk+write+known_if_redirected e81)", 2020-03-09T00:25:49.317 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:30.805722+0000", 2020-03-09T00:25:49.317 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 18.312769338999999, 2020-03-09T00:25:49.318 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038719972999999998, 2020-03-09T00:25:49.318 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.318 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.318 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.318 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.318 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 687 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805722+0000", 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.319 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.320 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.320 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.320 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805722+0000", 2020-03-09T00:25:49.320 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2537e-05 2020-03-09T00:25:49.320 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.320 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.320 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.321 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805795+0000", 2020-03-09T00:25:49.321 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8480000000000003e-06 2020-03-09T00:25:49.321 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.321 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.321 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.321 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805801+0000", 2020-03-09T00:25:49.321 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5707999999999999e-05 2020-03-09T00:25:49.322 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.322 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.322 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.322 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805817+0000", 2020-03-09T00:25:49.322 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9030000000000003e-06 2020-03-09T00:25:49.322 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.322 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805822+0000", 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018435500000000001 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806006+0000", 2020-03-09T00:25:49.323 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0726e-05 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806017+0000", 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020965160000000001 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.324 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.325 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808114+0000", 2020-03-09T00:25:49.325 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000119157 2020-03-09T00:25:49.325 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.325 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.325 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.325 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808233+0000", 2020-03-09T00:25:49.325 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020385461000000001 2020-03-09T00:25:49.326 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.326 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.326 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.326 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.828618+0000", 2020-03-09T00:25:49.326 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010904479 2020-03-09T00:25:49.326 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.326 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.327 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.327 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.839523+0000", 2020-03-09T00:25:49.327 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047344329999999997 2020-03-09T00:25:49.327 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.327 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.327 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:49.327 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.844257+0000", 2020-03-09T00:25:49.328 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018484999999999999 2020-03-09T00:25:49.328 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.328 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.328 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.328 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.844442+0000", 2020-03-09T00:25:49.328 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.038719972999999998 2020-03-09T00:25:49.328 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.329 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.329 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.329 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.329 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.329 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:688 3.9s0 3:91ab8712:::smithi10428818-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e81)", 2020-03-09T00:25:49.329 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:30.805872+0000", 2020-03-09T00:25:49.329 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 18.312619088999998, 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039786291000000001, 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 688 2020-03-09T00:25:49.330 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.331 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.331 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.331 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.331 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805872+0000", 2020-03-09T00:25:49.331 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.331 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.332 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.332 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.332 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805872+0000", 2020-03-09T00:25:49.332 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2169000000000001e-05 2020-03-09T00:25:49.332 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.332 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.332 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805924+0000", 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3419999999999996e-06 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805931+0000", 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.322e-05 2020-03-09T00:25:49.333 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.334 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.334 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.334 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805944+0000", 2020-03-09T00:25:49.334 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8330000000000001e-06 2020-03-09T00:25:49.334 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.334 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.334 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.335 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.805949+0000", 2020-03-09T00:25:49.335 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6842000000000006e-05 2020-03-09T00:25:49.335 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.335 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.335 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.335 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806046+0000", 2020-03-09T00:25:49.335 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5959999999999993e-06 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.806055+0000", 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002393306 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.336 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.337 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808449+0000", 2020-03-09T00:25:49.337 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0771e-05 2020-03-09T00:25:49.337 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.337 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.337 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:49.337 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.808489+0000", 2020-03-09T00:25:49.337 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.029870654999999999 2020-03-09T00:25:49.338 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.338 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.338 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.338 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.838360+0000", 2020-03-09T00:25:49.338 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015941600000000001 2020-03-09T00:25:49.338 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.338 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.339 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:49.339 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.838519+0000", 2020-03-09T00:25:49.339 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0068704389999999999 2020-03-09T00:25:49.339 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.339 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.339 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.339 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.845390+0000", 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7422000000000003e-05 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.845487+0000", 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017128 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.340 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.341 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.341 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:30.845659+0000", 2020-03-09T00:25:49.341 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039786291000000001 2020-03-09T00:25:49.341 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.341 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.341 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.341 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.342 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.342 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:961 3.9s0 3:9f137ce6:::smithi10428818-22:head [read 0~4407296,checksum] snapc 0=[] ondisk+read+known_if_redirected e96)", 2020-03-09T00:25:49.342 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:45.877163+0000", 2020-03-09T00:25:49.342 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.2413281060000001, 2020-03-09T00:25:49.342 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027797718999999999, 2020-03-09T00:25:49.342 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.342 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.343 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.343 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.343 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.343 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 961 2020-03-09T00:25:49.343 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.343 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.343 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877163+0000", 2020-03-09T00:25:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.344 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877163+0000", 2020-03-09T00:25:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7706000000000004e-05 2020-03-09T00:25:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877241+0000", 2020-03-09T00:25:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1709999999999999e-06 2020-03-09T00:25:49.345 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.346 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.346 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.346 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877247+0000", 2020-03-09T00:25:49.346 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1936000000000001e-05 2020-03-09T00:25:49.346 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.346 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.346 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.347 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877259+0000", 2020-03-09T00:25:49.347 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5310000000000003e-06 2020-03-09T00:25:49.347 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.347 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.347 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.347 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877264+0000", 2020-03-09T00:25:49.347 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7450000000000002e-05 2020-03-09T00:25:49.348 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.348 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.348 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.348 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877301+0000", 2020-03-09T00:25:49.348 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2969999999999992e-06 2020-03-09T00:25:49.348 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.348 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877310+0000", 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036065899999999998 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.880917+0000", 2020-03-09T00:25:49.349 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000146477 2020-03-09T00:25:49.350 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.350 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.350 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.350 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.881063+0000", 2020-03-09T00:25:49.350 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023897561000000001 2020-03-09T00:25:49.350 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.351 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.351 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.351 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.904961+0000", 2020-03-09T00:25:49.351 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027797718999999999 2020-03-09T00:25:49.351 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.351 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.351 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.352 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.352 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.352 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:965 3.9s0 3:9945b330:::smithi10428818-34:head [rollback 3a] snapc 3c=[3c,3b,3a,39,38] ondisk+write+known_if_redirected e96)", 2020-03-09T00:25:49.352 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:45.877762+0000", 2020-03-09T00:25:49.352 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 3.2407291059999999, 2020-03-09T00:25:49.352 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027669520999999999, 2020-03-09T00:25:49.352 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:49.353 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:49.353 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:49.353 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:49.353 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.353 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 965 2020-03-09T00:25:49.353 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.353 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:49.354 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.354 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:49.354 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877762+0000", 2020-03-09T00:25:49.354 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:49.354 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.354 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.354 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:49.355 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877762+0000", 2020-03-09T00:25:49.355 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3423000000000001e-05 2020-03-09T00:25:49.355 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.355 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.355 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:49.355 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877806+0000", 2020-03-09T00:25:49.356 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7769999999999999e-06 2020-03-09T00:25:49.356 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.356 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.356 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:49.356 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877811+0000", 2020-03-09T00:25:49.356 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1171e-05 2020-03-09T00:25:49.356 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.357 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.357 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:49.357 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877823+0000", 2020-03-09T00:25:49.357 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1200000000000004e-06 2020-03-09T00:25:49.357 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.357 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.358 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.358 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877827+0000", 2020-03-09T00:25:49.358 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1542000000000003e-05 2020-03-09T00:25:49.358 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.358 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.358 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.358 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877868+0000", 2020-03-09T00:25:49.359 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8439999999999993e-06 2020-03-09T00:25:49.359 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.359 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.359 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:49.359 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.877878+0000", 2020-03-09T00:25:49.359 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0036450430000000002 2020-03-09T00:25:49.359 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.360 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.360 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:49.360 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.881523+0000", 2020-03-09T00:25:49.360 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014482899999999999 2020-03-09T00:25:49.360 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.360 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.360 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:49.361 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.881668+0000", 2020-03-09T00:25:49.361 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023763771999999999 2020-03-09T00:25:49.361 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:49.361 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:49.361 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:49.362 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:45.905432+0000", 2020-03-09T00:25:49.362 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.027669520999999999 2020-03-09T00:25:49.362 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.362 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.362 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.362 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:49.362 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:49.363 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:49.366 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2020-03-09T00:25:49.367 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:25:49.367 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":99,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:48.879741+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-03-09T00:24:40.793412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":63,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:25:49.369 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json 2020-03-09T00:25:49.369 INFO:teuthology.orchestra.run.smithi148.stdout:osd.15: {} 2020-03-09T00:25:49.369 INFO:teuthology.orchestra.run.smithi148.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-03-09T00:25:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:osd.6: problem getting command descriptions from osd.6 2020-03-09T00:25:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:osd.8: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:osd.9: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:osd.10: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:osd.11: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:osd.12: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:osd.13: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.372 INFO:teuthology.orchestra.run.smithi148.stderr:osd.14: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.372 INFO:teuthology.orchestra.run.smithi148.stderr:osd.15: osd_enable_op_tracker = 'true' 2020-03-09T00:25:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-03-09T00:25:49.372 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-03-09T00:25:49.372 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:49.362+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:49.465 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:49.463+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:49.498 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:49.499 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:49.499 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:49.507 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2020-03-09T00:25:49.566 INFO:tasks.ceph.osd.7.smithi104.stderr:2020-03-09T00:25:49.564+0000 7f81b329d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28005) UID: 0 2020-03-09T00:25:49.622 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:49.622 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:49.623 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:49.623 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:49.623 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:49.631 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2020-03-09T00:25:49.664 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:25:49.665 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2020-03-09T00:25:49.667 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:49.666+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:49.674 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2020-03-09T00:25:48.883357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135725056,"num_objects":143,"num_object_clones":94,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":10,"num_read":463,"num_read_kb":207361,"num_write":796,"num_write_kb":163198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":29081600,"num_keys_recovered":3,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12158,"num_omap_keys":3,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1173,"ondisk_log_size":1173,"up":128,"acting":128,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":14913260,"kb_used_data":233084,"kb_used_omap":0,"kb_used_meta":14680064,"kb_avail":1306292500,"statfs":{"total":1352914698240,"available":1337643520000,"internally_reserved":15032385536,"allocated":238678016,"data_stored":353617375,"data_compressed":1937244,"data_compressed_allocated":4034560,"data_compressed_original":8069120,"omap_allocated":0,"internal_metadata":15032385536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10338304,"num_objects":18,"num_object_clones":18,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":39,"num_read_kb":24684,"num_write":118,"num_write_kb":9504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017718"},"pg_stats":[{"pgid":"3.f","version":"92'125","reported_seq":"366","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.899254+0000","last_change":"2020-03-09T00:25:40.858468+0000","last_active":"2020-03-09T00:25:44.899254+0000","last_peered":"2020-03-09T00:25:44.899254+0000","last_clean":"2020-03-09T00:25:44.899254+0000","last_became_active":"2020-03-09T00:24:58.785061+0000","last_became_peered":"2020-03-09T00:24:58.785061+0000","last_unstale":"2020-03-09T00:25:44.899254+0000","last_undegraded":"2020-03-09T00:25:44.899254+0000","last_fullsized":"2020-03-09T00:25:44.899254+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15417344,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27711,"num_write":66,"num_write_kb":18176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":4014080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,12,4,9,14,10,2],"acting":[11,12,4,9,14,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"c","length":"1"},{"start":"13","length":"1"},{"start":"1a","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"1"},{"start":"2a","length":"3"},{"start":"32","length":"2"},{"start":"35","length":"1"}]},{"pgid":"3.e","version":"88'146","reported_seq":"351","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.925646+0000","last_change":"2020-03-09T00:25:41.088412+0000","last_active":"2020-03-09T00:25:44.925646+0000","last_peered":"2020-03-09T00:25:44.925646+0000","last_clean":"2020-03-09T00:25:44.925646+0000","last_became_active":"2020-03-09T00:24:43.219818+0000","last_became_peered":"2020-03-09T00:24:43.219818+0000","last_unstale":"2020-03-09T00:25:44.925646+0000","last_undegraded":"2020-03-09T00:25:44.925646+0000","last_fullsized":"2020-03-09T00:25:44.925646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":146,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2654208,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":25556,"num_write":51,"num_write_kb":13824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,11,2,0,3],"acting":[8,9,1,11,2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"c","length":"1"},{"start":"13","length":"1"},{"start":"1a","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"1"},{"start":"2a","length":"3"},{"start":"32","length":"2"},{"start":"35","length":"1"}]},{"pgid":"3.d","version":"0'0","reported_seq":"142","reported_epoch":"95","state":"active+undersized","last_fresh":"2020-03-09T00:25:44.873732+0000","last_change":"2020-03-09T00:25:17.773033+0000","last_active":"2020-03-09T00:25:44.873732+0000","last_peered":"2020-03-09T00:25:44.873732+0000","last_clean":"2020-03-09T00:25:16.317479+0000","last_became_active":"2020-03-09T00:25:17.773033+0000","last_became_peered":"2020-03-09T00:25:17.773033+0000","last_unstale":"2020-03-09T00:25:44.873732+0000","last_undegraded":"2020-03-09T00:25:44.873732+0000","last_fullsized":"2020-03-09T00:25:17.766872+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0,12,10,2,2147483647],"acting":[7,3,0,12,10,2,2147483647],"avail_no_missing":["7(0)","0(2)","2(5)","3(1)","10(4)","12(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"88'92","reported_seq":"251","reported_epoch":"95","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:44.916687+0000","last_change":"2020-03-09T00:25:17.780040+0000","last_active":"2020-03-09T00:25:44.916687+0000","last_peered":"2020-03-09T00:25:44.916687+0000","last_clean":"2020-03-09T00:25:16.114410+0000","last_became_active":"2020-03-09T00:25:17.780040+0000","last_became_peered":"2020-03-09T00:25:17.780040+0000","last_unstale":"2020-03-09T00:25:44.916687+0000","last_undegraded":"2020-03-09T00:25:17.769871+0000","last_fullsized":"2020-03-09T00:25:17.769497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":15794176,"num_objects":18,"num_object_clones":13,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":44,"num_read_kb":11889,"num_write":46,"num_write_kb":15424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,14,12,4,2,2147483647],"acting":[11,10,14,12,4,2,2147483647],"avail_no_missing":["11(0)","2(5)","4(4)","10(1)","12(3)","14(2)"],"object_location_counts":[{"shards":"2(5),4(4),10(1),11(0),12(3),14(2)","objects":18}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"94'64","reported_seq":"203","reported_epoch":"95","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:44.896112+0000","last_change":"2020-03-09T00:25:17.774494+0000","last_active":"2020-03-09T00:25:44.896112+0000","last_peered":"2020-03-09T00:25:44.896112+0000","last_clean":"2020-03-09T00:25:16.088901+0000","last_became_active":"2020-03-09T00:25:17.774494+0000","last_became_peered":"2020-03-09T00:25:17.774494+0000","last_unstale":"2020-03-09T00:25:44.896112+0000","last_undegraded":"2020-03-09T00:25:17.767167+0000","last_fullsized":"2020-03-09T00:25:17.766734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":4308992,"num_objects":10,"num_object_clones":7,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":17,"num_read_kb":3798,"num_write":59,"num_write_kb":5584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2147483647,1,9,8,7,14],"acting":[10,2147483647,1,9,8,7,14],"avail_no_missing":["10(0)","1(2)","7(5)","8(4)","9(3)","14(6)"],"object_location_counts":[{"shards":"1(2),7(5),8(4),9(3),10(0),14(6)","objects":10}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.a","version":"94'43","reported_seq":"233","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.901490+0000","last_change":"2020-03-09T00:25:40.859914+0000","last_active":"2020-03-09T00:25:44.901490+0000","last_peered":"2020-03-09T00:25:44.901490+0000","last_clean":"2020-03-09T00:25:44.901490+0000","last_became_active":"2020-03-09T00:24:43.086461+0000","last_became_peered":"2020-03-09T00:24:43.086461+0000","last_unstale":"2020-03-09T00:25:44.901490+0000","last_undegraded":"2020-03-09T00:25:44.901490+0000","last_fullsized":"2020-03-09T00:25:44.901490+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4554752,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":14,"num_write_kb":5600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,8,15,11],"acting":[3,7,1,9,8,15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"13","length":"1"},{"start":"1a","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"1"},{"start":"2a","length":"3"},{"start":"32","length":"2"},{"start":"35","length":"1"}]},{"pgid":"3.9","version":"96'175","reported_seq":"335","reported_epoch":"96","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:45.904725+0000","last_change":"2020-03-09T00:25:17.777991+0000","last_active":"2020-03-09T00:25:45.904725+0000","last_peered":"2020-03-09T00:25:45.904725+0000","last_clean":"2020-03-09T00:25:15.780057+0000","last_became_active":"2020-03-09T00:25:17.777991+0000","last_became_peered":"2020-03-09T00:25:17.777991+0000","last_unstale":"2020-03-09T00:25:45.904725+0000","last_undegraded":"2020-03-09T00:25:17.769806+0000","last_fullsized":"2020-03-09T00:25:17.766736+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":175,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":21331968,"num_objects":28,"num_object_clones":21,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":2,"num_read":72,"num_read_kb":36084,"num_write":157,"num_write_kb":23360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,14,4,15,10,2147483647],"acting":[12,1,14,4,15,10,2147483647],"avail_no_missing":["12(0)","1(1)","4(3)","10(5)","14(2)","15(4)"],"object_location_counts":[{"shards":"1(1),4(3),10(5),12(0),14(2),15(4)","objects":28}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"88'60","reported_seq":"233","reported_epoch":"95","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:44.870438+0000","last_change":"2020-03-09T00:25:17.776156+0000","last_active":"2020-03-09T00:25:44.870438+0000","last_peered":"2020-03-09T00:25:44.870438+0000","last_clean":"2020-03-09T00:25:16.088428+0000","last_became_active":"2020-03-09T00:25:17.776156+0000","last_became_peered":"2020-03-09T00:25:17.776156+0000","last_unstale":"2020-03-09T00:25:44.870438+0000","last_undegraded":"2020-03-09T00:25:17.767190+0000","last_fullsized":"2020-03-09T00:25:17.766818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":12943360,"num_objects":10,"num_object_clones":7,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":26350,"num_write":35,"num_write_kb":12640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,0,2147483647,14,15,1],"acting":[4,10,0,2147483647,14,15,1],"avail_no_missing":["4(0)","0(2)","1(6)","10(1)","14(4)","15(5)"],"object_location_counts":[{"shards":"0(2),1(6),4(0),10(1),14(4),15(5)","objects":10}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"36","reported_epoch":"95","state":"active+undersized","last_fresh":"2020-03-09T00:25:44.900144+0000","last_change":"2020-03-09T00:25:17.774328+0000","last_active":"2020-03-09T00:25:44.900144+0000","last_peered":"2020-03-09T00:25:44.900144+0000","last_clean":"2020-03-09T00:24:26.903022+0000","last_became_active":"2020-03-09T00:25:17.774328+0000","last_became_peered":"2020-03-09T00:25:17.774328+0000","last_unstale":"2020-03-09T00:25:44.900144+0000","last_undegraded":"2020-03-09T00:25:44.900144+0000","last_fullsized":"2020-03-09T00:25:17.772196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12],"acting":[12],"avail_no_missing":["12"],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.5","version":"94'39","reported_seq":"227","reported_epoch":"95","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:44.874391+0000","last_change":"2020-03-09T00:25:17.780474+0000","last_active":"2020-03-09T00:25:44.874391+0000","last_peered":"2020-03-09T00:25:44.874391+0000","last_clean":"2020-03-09T00:25:16.088513+0000","last_became_active":"2020-03-09T00:25:17.780474+0000","last_became_peered":"2020-03-09T00:25:17.780474+0000","last_unstale":"2020-03-09T00:25:44.874391+0000","last_undegraded":"2020-03-09T00:25:17.766960+0000","last_fullsized":"2020-03-09T00:25:17.766520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":6127616,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":13545,"num_write":21,"num_write_kb":5984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2147483647,14,3,7,11,8],"acting":[15,2147483647,14,3,7,11,8],"avail_no_missing":["15(0)","3(3)","7(4)","8(6)","11(5)","14(2)"],"object_location_counts":[{"shards":"3(3),7(4),8(6),11(5),14(2),15(0)","objects":5}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"75","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.901324+0000","last_change":"2020-03-09T00:24:27.942443+0000","last_active":"2020-03-09T00:25:44.901324+0000","last_peered":"2020-03-09T00:25:44.901324+0000","last_clean":"2020-03-09T00:25:44.901324+0000","last_became_active":"2020-03-09T00:24:27.942050+0000","last_became_peered":"2020-03-09T00:24:27.942050+0000","last_unstale":"2020-03-09T00:25:44.901324+0000","last_undegraded":"2020-03-09T00:25:44.901324+0000","last_fullsized":"2020-03-09T00:25:44.901324+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"95'57","reported_seq":"249","reported_epoch":"95","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:44.909078+0000","last_change":"2020-03-09T00:25:17.775200+0000","last_active":"2020-03-09T00:25:44.909078+0000","last_peered":"2020-03-09T00:25:44.909078+0000","last_clean":"2020-03-09T00:25:16.063325+0000","last_became_active":"2020-03-09T00:25:17.775200+0000","last_became_peered":"2020-03-09T00:25:17.775200+0000","last_unstale":"2020-03-09T00:25:44.909078+0000","last_undegraded":"2020-03-09T00:25:17.767682+0000","last_fullsized":"2020-03-09T00:25:17.767309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":4407296,"num_objects":12,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":33,"num_read_kb":11909,"num_write":62,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3620864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,15,4,2147483647,3],"acting":[0,7,2,15,4,2147483647,3],"avail_no_missing":["0(0)","2(2)","3(6)","4(4)","7(1)","15(3)"],"object_location_counts":[{"shards":"0(0),2(2),3(6),4(4),7(1),15(3)","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"62","reported_epoch":"96","state":"active+clean","last_fresh":"2020-03-09T00:25:45.871789+0000","last_change":"2020-03-09T00:25:38.217575+0000","last_active":"2020-03-09T00:25:45.871789+0000","last_peered":"2020-03-09T00:25:45.871789+0000","last_clean":"2020-03-09T00:25:45.871789+0000","last_became_active":"2020-03-09T00:25:01.766731+0000","last_became_peered":"2020-03-09T00:25:01.766731+0000","last_unstale":"2020-03-09T00:25:45.871789+0000","last_undegraded":"2020-03-09T00:25:45.871789+0000","last_fullsized":"2020-03-09T00:25:45.871789+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:25:38.217513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:25:38.217513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4],"acting":[14,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"96'94","reported_seq":"290","reported_epoch":"96","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:45.898773+0000","last_change":"2020-03-09T00:25:17.777748+0000","last_active":"2020-03-09T00:25:45.898773+0000","last_peered":"2020-03-09T00:25:45.898773+0000","last_clean":"2020-03-09T00:25:15.779988+0000","last_became_active":"2020-03-09T00:25:17.777748+0000","last_became_peered":"2020-03-09T00:25:17.777748+0000","last_unstale":"2020-03-09T00:25:45.898773+0000","last_undegraded":"2020-03-09T00:25:17.767130+0000","last_fullsized":"2020-03-09T00:25:17.766750+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":7405568,"num_objects":13,"num_object_clones":9,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":44,"num_read_kb":17972,"num_write":56,"num_write_kb":9472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3768320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,11,15,1,10,0,2147483647],"acting":[12,11,15,1,10,0,2147483647],"avail_no_missing":["12(0)","0(5)","1(3)","10(4)","11(1)","15(2)"],"object_location_counts":[{"shards":"0(5),1(3),10(4),11(1),12(0),15(2)","objects":13}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"55","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.874035+0000","last_change":"2020-03-09T00:25:01.776912+0000","last_active":"2020-03-09T00:25:44.874035+0000","last_peered":"2020-03-09T00:25:44.874035+0000","last_clean":"2020-03-09T00:25:44.874035+0000","last_became_active":"2020-03-09T00:25:01.776519+0000","last_became_peered":"2020-03-09T00:25:01.776519+0000","last_unstale":"2020-03-09T00:25:44.874035+0000","last_undegraded":"2020-03-09T00:25:44.874035+0000","last_fullsized":"2020-03-09T00:25:44.874035+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8],"acting":[15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"65'6","reported_seq":"92","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.873316+0000","last_change":"2020-03-09T00:25:15.037301+0000","last_active":"2020-03-09T00:25:44.873316+0000","last_peered":"2020-03-09T00:25:44.873316+0000","last_clean":"2020-03-09T00:25:44.873316+0000","last_became_active":"2020-03-09T00:24:58.776593+0000","last_became_peered":"2020-03-09T00:24:58.776593+0000","last_unstale":"2020-03-09T00:25:44.873316+0000","last_undegraded":"2020-03-09T00:25:44.873316+0000","last_fullsized":"2020-03-09T00:25:44.873316+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":3,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2020-03-09T00:25:15.037239+0000","last_deep_scrub":"65'6","last_deep_scrub_stamp":"2020-03-09T00:25:15.037239+0000","last_clean_scrub_stamp":"2020-03-09T00:25:15.037239+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":3,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12158,"num_omap_keys":3,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"95'81","reported_seq":"289","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.915955+0000","last_change":"2020-03-09T00:25:40.848052+0000","last_active":"2020-03-09T00:25:44.915955+0000","last_peered":"2020-03-09T00:25:44.915955+0000","last_clean":"2020-03-09T00:25:44.915955+0000","last_became_active":"2020-03-09T00:24:58.777683+0000","last_became_peered":"2020-03-09T00:24:58.777683+0000","last_unstale":"2020-03-09T00:25:44.915955+0000","last_undegraded":"2020-03-09T00:25:44.915955+0000","last_fullsized":"2020-03-09T00:25:44.915955+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6389760,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":6732,"num_write":71,"num_write_kb":10384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,12,3,4,15,1,11],"acting":[9,12,3,4,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"c","length":"1"},{"start":"13","length":"1"},{"start":"1a","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"1"},{"start":"2a","length":"3"},{"start":"32","length":"2"},{"start":"35","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":"78","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.901250+0000","last_change":"2020-03-09T00:25:35.882732+0000","last_active":"2020-03-09T00:25:44.901250+0000","last_peered":"2020-03-09T00:25:44.901250+0000","last_clean":"2020-03-09T00:25:44.901250+0000","last_became_active":"2020-03-09T00:24:27.940904+0000","last_became_peered":"2020-03-09T00:24:27.940904+0000","last_unstale":"2020-03-09T00:25:44.901250+0000","last_undegraded":"2020-03-09T00:25:44.901250+0000","last_fullsized":"2020-03-09T00:25:44.901250+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:25:35.882647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:25:35.882647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"94'81","reported_seq":"315","reported_epoch":"95","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:44.899089+0000","last_change":"2020-03-09T00:25:17.779414+0000","last_active":"2020-03-09T00:25:44.899089+0000","last_peered":"2020-03-09T00:25:44.899089+0000","last_clean":"2020-03-09T00:25:14.798877+0000","last_became_active":"2020-03-09T00:25:17.779414+0000","last_became_peered":"2020-03-09T00:25:17.779414+0000","last_unstale":"2020-03-09T00:25:44.899089+0000","last_undegraded":"2020-03-09T00:25:17.768694+0000","last_fullsized":"2020-03-09T00:25:17.768294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":9175040,"num_objects":12,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":33,"num_read_kb":11917,"num_write":51,"num_write_kb":10720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":9895936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,11,9,1,12,0,4],"acting":[2147483647,11,9,1,12,0,4],"avail_no_missing":["11(1)","0(5)","1(3)","4(6)","9(2)","12(4)"],"object_location_counts":[{"shards":"0(5),1(3),4(6),9(2),11(1),12(4)","objects":12}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"76","reported_epoch":"95","state":"active+clean","last_fresh":"2020-03-09T00:25:44.925658+0000","last_change":"2020-03-09T00:24:27.925113+0000","last_active":"2020-03-09T00:25:44.925658+0000","last_peered":"2020-03-09T00:25:44.925658+0000","last_clean":"2020-03-09T00:25:44.925658+0000","last_became_active":"2020-03-09T00:24:27.924718+0000","last_became_peered":"2020-03-09T00:24:27.924718+0000","last_unstale":"2020-03-09T00:25:44.925658+0000","last_undegraded":"2020-03-09T00:25:44.925658+0000","last_fullsized":"2020-03-09T00:25:44.925658+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'14","reported_seq":"194","reported_epoch":"95","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:44.904716+0000","last_change":"2020-03-09T00:25:17.778237+0000","last_active":"2020-03-09T00:25:44.904716+0000","last_peered":"2020-03-09T00:25:44.904716+0000","last_clean":"2020-03-09T00:25:16.089071+0000","last_became_active":"2020-03-09T00:25:17.778237+0000","last_became_peered":"2020-03-09T00:25:17.778237+0000","last_unstale":"2020-03-09T00:25:44.904716+0000","last_undegraded":"2020-03-09T00:25:17.768799+0000","last_fullsized":"2020-03-09T00:25:17.768422+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":3309568,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":1157,"num_write":8,"num_write_kb":3808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,12,14,1,2147483647,11,0],"acting":[10,12,14,1,2147483647,11,0],"avail_no_missing":["10(0)","0(6)","1(3)","11(5)","12(1)","14(2)"],"object_location_counts":[{"shards":"0(6),1(3),10(0),11(5),12(1),14(2)","objects":4}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"94'45","reported_seq":"228","reported_epoch":"96","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:45.872068+0000","last_change":"2020-03-09T00:25:17.775455+0000","last_active":"2020-03-09T00:25:45.872068+0000","last_peered":"2020-03-09T00:25:45.872068+0000","last_clean":"2020-03-09T00:25:16.089536+0000","last_became_active":"2020-03-09T00:25:17.775455+0000","last_became_peered":"2020-03-09T00:25:17.775455+0000","last_unstale":"2020-03-09T00:25:45.872068+0000","last_undegraded":"2020-03-09T00:25:17.766364+0000","last_fullsized":"2020-03-09T00:25:17.765992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":9469952,"num_objects":5,"num_object_clones":4,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":438,"num_write":41,"num_write_kb":11424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1671168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,9,4,2,2147483647,11],"acting":[14,8,9,4,2,2147483647,11],"avail_no_missing":["14(0)","2(4)","4(3)","8(1)","9(2)","11(6)"],"object_location_counts":[{"shards":"2(4),4(3),8(1),9(2),11(6),14(0)","objects":5}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"77","reported_epoch":"96","state":"active+clean","last_fresh":"2020-03-09T00:25:45.872383+0000","last_change":"2020-03-09T00:24:27.941155+0000","last_active":"2020-03-09T00:25:45.872383+0000","last_peered":"2020-03-09T00:25:45.872383+0000","last_clean":"2020-03-09T00:25:45.872383+0000","last_became_active":"2020-03-09T00:24:27.940788+0000","last_became_peered":"2020-03-09T00:24:27.940788+0000","last_unstale":"2020-03-09T00:25:45.872383+0000","last_undegraded":"2020-03-09T00:25:45.872383+0000","last_fullsized":"2020-03-09T00:25:45.872383+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"88","reported_epoch":"95","state":"active+undersized","last_fresh":"2020-03-09T00:25:44.874170+0000","last_change":"2020-03-09T00:25:17.767698+0000","last_active":"2020-03-09T00:25:44.874170+0000","last_peered":"2020-03-09T00:25:44.874170+0000","last_clean":"2020-03-09T00:25:16.088554+0000","last_became_active":"2020-03-09T00:25:17.767698+0000","last_became_peered":"2020-03-09T00:25:17.767698+0000","last_unstale":"2020-03-09T00:25:44.874170+0000","last_undegraded":"2020-03-09T00:25:44.874170+0000","last_fullsized":"2020-03-09T00:25:17.766737+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:26.903022+0000","last_clean_scrub_stamp":"2020-03-09T00:24:26.903022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15],"acting":[15],"avail_no_missing":["15"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"95'51","reported_seq":"243","reported_epoch":"96","state":"active+undersized+degraded","last_fresh":"2020-03-09T00:25:45.872024+0000","last_change":"2020-03-09T00:25:17.776403+0000","last_active":"2020-03-09T00:25:45.872024+0000","last_peered":"2020-03-09T00:25:45.872024+0000","last_clean":"2020-03-09T00:25:16.089526+0000","last_became_active":"2020-03-09T00:25:17.776403+0000","last_became_peered":"2020-03-09T00:25:17.776403+0000","last_unstale":"2020-03-09T00:25:45.872024+0000","last_undegraded":"2020-03-09T00:25:17.766554+0000","last_fullsized":"2020-03-09T00:25:17.766176+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-09T00:24:41.780905+0000","last_clean_scrub_stamp":"2020-03-09T00:24:41.780905+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":12435456,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":31,"num_read_kb":12301,"num_write":55,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,1,15,2147483647,7,9],"acting":[14,0,1,15,2147483647,7,9],"avail_no_missing":["14(0)","0(1)","1(2)","7(5)","9(6)","15(3)"],"object_location_counts":[{"shards":"0(1),1(2),7(5),9(6),14(0),15(3)","objects":9}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":135725056,"num_objects":140,"num_object_clones":94,"num_object_copies":980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":10,"num_read":463,"num_read_kb":207361,"num_write":793,"num_write_kb":163184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":29081600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229720064,"data_stored":350756864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1167,"ondisk_log_size":1167,"up":112,"acting":112,"num_store_stats":16},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":3,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12158,"num_omap_keys":3,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":2,"acting":2,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":12}],"osd_stats":[{"osd":15,"up_from":24,"seq":103079215124,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1067968,"kb_used_data":19384,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93303872,"statfs":{"total":96636764160,"available":95543164928,"internally_reserved":1073741824,"allocated":19849216,"data_stored":31861555,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":7.7910000000000004,"5min":7.7910000000000004,"15min":7.7910000000000004},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":17.603000000000002,"5min":17.603000000000002,"15min":17.603000000000002},"last":0.79900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":10.965,"5min":10.965,"15min":10.965},"last":1.0660000000000001},{"interface":"front","average":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"min":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"max":{"1min":14.848000000000001,"5min":14.848000000000001,"15min":14.848000000000001},"last":1.3260000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":7,"last update":"Mon Mar 9 00:25:44 2020","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.357}]},{"osd":8,"last update":"Mon Mar 9 00:25:44 2020","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2,"5min":2,"15min":2},"last":1.462},{"interface":"front","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":1.133,"5min":1.133,"15min":1.133},"max":{"1min":1.921,"5min":1.921,"15min":1.921},"last":1.4470000000000001}]},{"osd":9,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":2.6099999999999999},"min":{"1min":1.325,"5min":1.325,"15min":1.325},"max":{"1min":17.904,"5min":17.904,"15min":17.904},"last":1.5489999999999999},{"interface":"front","average":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"min":{"1min":1.107,"5min":1.107,"15min":1.107},"max":{"1min":14.898,"5min":14.898,"15min":14.898},"last":1.7}]},{"osd":10,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":2.347,"5min":2.347,"15min":2.347},"min":{"1min":1.22,"5min":1.22,"15min":1.22},"max":{"1min":14.616,"5min":14.616,"15min":14.616},"last":1.53},{"interface":"front","average":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"min":{"1min":1.331,"5min":1.331,"15min":1.331},"max":{"1min":16.292999999999999,"5min":16.292999999999999,"15min":16.292999999999999},"last":1.6220000000000001}]},{"osd":11,"last update":"Mon Mar 9 00:25:44 2020","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":2.234,"5min":2.234,"15min":2.234},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"min":{"1min":1.397,"5min":1.397,"15min":1.397},"max":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"last":1.6020000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":14,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"min":{"1min":1.131,"5min":1.131,"15min":1.131},"max":{"1min":9.5069999999999997,"5min":9.5069999999999997,"15min":9.5069999999999997},"last":1.4850000000000001},{"interface":"front","average":{"1min":2.411,"5min":2.411,"15min":2.411},"min":{"1min":1.534,"5min":1.534,"15min":1.534},"max":{"1min":12.859999999999999,"5min":12.859999999999999,"15min":12.859999999999999},"last":1.6799999999999999}]}]},{"osd":14,"up_from":24,"seq":103079215123,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1074204,"kb_used_data":25620,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297636,"statfs":{"total":96636764160,"available":95536779264,"internally_reserved":1073741824,"allocated":26234880,"data_stored":40520499,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,7,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":6.2229999999999999,"5min":6.2229999999999999,"15min":6.2229999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":3.2850000000000001,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.29}]},{"osd":1,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":6.7320000000000002,"5min":6.7320000000000002,"15min":6.7320000000000002},"last":1.631}]},{"osd":2,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.9169999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.71299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":4,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":3.5089999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":6.6239999999999997,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":0.98599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":7,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"min":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"max":{"1min":6.3879999999999999,"5min":6.3879999999999999,"15min":6.3879999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"min":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"max":{"1min":7.7279999999999998,"5min":7.7279999999999998,"15min":7.7279999999999998},"last":1.018}]},{"osd":8,"last update":"Mon Mar 9 00:25:43 2020","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":2.282,"5min":2.282,"15min":2.282},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"min":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"max":{"1min":6.8630000000000004,"5min":6.8630000000000004,"15min":6.8630000000000004},"last":1.1419999999999999}]},{"osd":9,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"min":{"1min":1.5,"5min":1.5,"15min":1.5},"max":{"1min":3.0449999999999999,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.337},{"interface":"front","average":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"min":{"1min":1.29,"5min":1.29,"15min":1.29},"max":{"1min":7.1079999999999997,"5min":7.1079999999999997,"15min":7.1079999999999997},"last":1.474}]},{"osd":11,"last update":"Mon Mar 9 00:25:43 2020","interfaces":[{"interface":"back","average":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"min":{"1min":1.101,"5min":1.101,"15min":1.101},"max":{"1min":8.9139999999999997,"5min":8.9139999999999997,"15min":8.9139999999999997},"last":1.163},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.0419999999999998,"15min":2.0419999999999998},"min":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"max":{"1min":3.5840000000000001,"5min":3.5840000000000001,"15min":3.5840000000000001},"last":1.581}]},{"osd":13,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"min":{"1min":1.393,"5min":1.393,"15min":1.393},"max":{"1min":7.8810000000000002,"5min":7.8810000000000002,"15min":7.8810000000000002},"last":1.452},{"interface":"front","average":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"min":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"max":{"1min":3.2349999999999999,"5min":3.2349999999999999,"15min":3.2349999999999999},"last":1.6599999999999999}]},{"osd":15,"last update":"Mon Mar 9 00:25:26 2020","interfaces":[{"interface":"back","average":{"1min":2.226,"5min":2.226,"15min":2.226},"min":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"max":{"1min":7.8070000000000004,"5min":7.8070000000000004,"15min":7.8070000000000004},"last":1.3220000000000001},{"interface":"front","average":{"1min":2.141,"5min":2.141,"15min":2.141},"min":{"1min":1.304,"5min":1.304,"15min":1.304},"max":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":1.8029999999999999}]}]},{"osd":13,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":12,"up_from":24,"seq":103079215124,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1068756,"kb_used_data":20172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93303084,"statfs":{"total":96636764160,"available":95542358016,"internally_reserved":1073741824,"allocated":20656128,"data_stored":30616371,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.98299999999999998}]},{"osd":1,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.5780000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Mon Mar 9 00:25:42 2020","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":1.157},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.80700000000000005}]},{"osd":5,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.466},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.9369999999999998,"5min":2.9369999999999998,"15min":2.9369999999999998},"last":0.96399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.073}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.55}]},{"osd":10,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.5529999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":1.728}]},{"osd":11,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.825,"15min":1.825},"min":{"1min":1.244,"5min":1.244,"15min":1.244},"max":{"1min":3.5249999999999999,"5min":3.5249999999999999,"15min":3.5249999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.284}]},{"osd":13,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"min":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.651},{"interface":"front","average":{"1min":1.823,"5min":1.823,"15min":1.823},"min":{"1min":1.129,"5min":1.129,"15min":1.129},"max":{"1min":3.6699999999999999,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.4259999999999999}]},{"osd":14,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"min":{"1min":1.413,"5min":1.413,"15min":1.413},"max":{"1min":3.8079999999999998,"5min":3.8079999999999998,"15min":3.8079999999999998},"last":1.53},{"interface":"front","average":{"1min":1.74,"5min":1.74,"15min":1.74},"min":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"max":{"1min":2.7149999999999999,"5min":2.7149999999999999,"15min":2.7149999999999999},"last":1.99}]},{"osd":15,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.861,"5min":1.861,"15min":1.861},"min":{"1min":1.22,"5min":1.22,"15min":1.22},"max":{"1min":2.8860000000000001,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.712,"5min":1.712,"15min":1.712},"min":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.8779999999999999}]}]},{"osd":11,"up_from":24,"seq":103079215124,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069264,"kb_used_data":20680,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93302576,"statfs":{"total":96636764160,"available":95541837824,"internally_reserved":1073741824,"allocated":21176320,"data_stored":30784307,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":2.2909999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.93999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":2,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.093}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":4,"last update":"Mon Mar 9 00:25:42 2020","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.84299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":9,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.268},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.399}]},{"osd":10,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":2.0609999999999999,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":1.046,"5min":1.046,"15min":1.046},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":1.1890000000000001}]},{"osd":12,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.784,"5min":1.784,"15min":1.784},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.458}]},{"osd":13,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.677,"5min":2.677,"15min":2.677},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.556}]},{"osd":14,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.829,"5min":1.829,"15min":1.829},"min":{"1min":1.327,"5min":1.327,"15min":1.327},"max":{"1min":2.8399999999999999,"5min":2.8399999999999999,"15min":2.8399999999999999},"last":1.54},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":1.343}]},{"osd":15,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":3.1099999999999999,"5min":3.1099999999999999,"15min":3.1099999999999999},"last":1.6160000000000001}]}]},{"osd":9,"up_from":24,"seq":103079215124,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1065384,"kb_used_data":16800,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306456,"statfs":{"total":96636764160,"available":95545810944,"internally_reserved":1073741824,"allocated":17203200,"data_stored":24632115,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":3,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.137,"15min":1.137},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.5309999999999999}]},{"osd":4,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.431},{"interface":"front","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.96199999999999997}]},{"osd":8,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.853},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":1.7829999999999999}]},{"osd":10,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":2.2080000000000002},{"interface":"front","average":{"1min":1.522,"5min":1.522,"15min":1.522},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.103}]},{"osd":11,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.8820000000000001,"5min":2.8820000000000001,"15min":2.8820000000000001},"last":2.5150000000000001},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"min":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.9339999999999999}]},{"osd":12,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.333},{"interface":"front","average":{"1min":1.806,"5min":1.806,"15min":1.806},"min":{"1min":1.294,"5min":1.294,"15min":1.294},"max":{"1min":2.7959999999999998,"5min":2.7959999999999998,"15min":2.7959999999999998},"last":2.3039999999999998}]},{"osd":13,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.794,"15min":1.794},"min":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":1.538},{"interface":"front","average":{"1min":1.637,"5min":1.637,"15min":1.637},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":1.3169999999999999}]},{"osd":14,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"min":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":2.25},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"max":{"1min":3.0910000000000002,"5min":3.0910000000000002,"15min":3.0910000000000002},"last":1.4690000000000001}]},{"osd":15,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"min":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":2.6989999999999998},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"min":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"max":{"1min":2.6429999999999998,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":2.4449999999999998}]}]},{"osd":8,"up_from":24,"seq":103079215123,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1056280,"kb_used_data":7696,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93315560,"statfs":{"total":96636764160,"available":95555133440,"internally_reserved":1073741824,"allocated":7880704,"data_stored":11279155,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,7,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:43 2020","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.85999999999999999}]},{"osd":1,"last update":"Mon Mar 9 00:25:20 2020","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":1.048},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.78800000000000003}]},{"osd":2,"last update":"Mon Mar 9 00:25:29 2020","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":1.46},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.5309999999999999}]},{"osd":3,"last update":"Mon Mar 9 00:25:43 2020","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":1.069},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.98399999999999999}]},{"osd":7,"last update":"Mon Mar 9 00:25:20 2020","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":3.6360000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.986,"5min":1.986,"15min":1.986},"last":4.2069999999999999}]},{"osd":9,"last update":"Mon Mar 9 00:25:20 2020","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.383}]},{"osd":10,"last update":"Mon Mar 9 00:25:20 2020","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":1.083,"5min":1.083,"15min":1.083},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.319},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.502}]},{"osd":11,"last update":"Mon Mar 9 00:25:20 2020","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":1.264},{"interface":"front","average":{"1min":1.589,"5min":1.589,"15min":1.589},"min":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"max":{"1min":2.395,"5min":2.395,"15min":2.395},"last":3.5649999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.353}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":14,"last update":"Mon Mar 9 00:25:20 2020","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":1.115,"5min":1.115,"15min":1.115},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":1.605},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.3460000000000001}]},{"osd":15,"last update":"Mon Mar 9 00:25:20 2020","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":2.4740000000000002,"5min":2.4740000000000002,"15min":2.4740000000000002},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.724,"5min":1.724,"15min":1.724},"min":{"1min":1.321,"5min":1.321,"15min":1.321},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.4279999999999999}]}]},{"osd":7,"up_from":24,"seq":103079215124,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1057432,"kb_used_data":8848,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93314408,"statfs":{"total":96636764160,"available":95553953792,"internally_reserved":1073741824,"allocated":9060352,"data_stored":14256947,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.24}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":2,"last update":"Mon Mar 9 00:25:44 2020","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":1.089},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":3.0550000000000002,"5min":3.0550000000000002,"15min":3.0550000000000002},"last":1.1859999999999999}]},{"osd":3,"last update":"Mon Mar 9 00:25:44 2020","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.153}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":8,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":2.661,"5min":2.661,"15min":2.661},"last":1.2909999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":10,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":1.73,"15min":1.73},"min":{"1min":1.04,"5min":1.04,"15min":1.04},"max":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":3.8700000000000001},"last":1.819},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"min":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"max":{"1min":2.242,"5min":2.242,"15min":2.242},"last":1.0720000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]},{"osd":12,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":1.766,"15min":1.766},"min":{"1min":1.236,"5min":1.236,"15min":1.236},"max":{"1min":3.3239999999999998,"5min":3.3239999999999998,"15min":3.3239999999999998},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":1.2729999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":14,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.925,"15min":1.925},"min":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":1.571},{"interface":"front","average":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"min":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":4.9050000000000002,"5min":4.9050000000000002,"15min":4.9050000000000002},"last":1.353}]},{"osd":15,"last update":"Mon Mar 9 00:25:24 2020","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.968,"15min":1.968},"min":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"max":{"1min":5.0650000000000004,"5min":5.0650000000000004,"15min":5.0650000000000004},"last":1.5309999999999999},{"interface":"front","average":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"min":{"1min":1.419,"5min":1.419,"15min":1.419},"max":{"1min":5.5940000000000003,"5min":5.5940000000000003,"15min":5.5940000000000003},"last":2.0129999999999999}]}]},{"osd":6,"up_from":24,"seq":103079215117,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1061736,"kb_used_data":13152,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310104,"statfs":{"total":96636764160,"available":95549546496,"internally_reserved":1073741824,"allocated":13467648,"data_stored":17433856,"data_compressed":73901,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,4,5,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8560000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]}]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215123,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1062436,"kb_used_data":13852,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93309404,"statfs":{"total":96636764160,"available":95548829696,"internally_reserved":1073741824,"allocated":14184448,"data_stored":23350067,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.0659999999999998,"5min":3.0659999999999998,"15min":3.0659999999999998},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.77300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":3,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":3.7330000000000001,"5min":3.7330000000000001,"15min":3.7330000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":3.4249999999999998,"5min":3.4249999999999998,"15min":3.4249999999999998},"last":1.3500000000000001}]},{"osd":4,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.9020000000000001,"5min":2.9020000000000001,"15min":2.9020000000000001},"last":1.194},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.0429999999999999}]},{"osd":5,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.48,"5min":2.48,"15min":2.48},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":3.9020000000000001},"last":0.80800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1960000000000002}]},{"osd":7,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":4.3959999999999999,"5min":4.3959999999999999,"15min":4.3959999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":4.0570000000000004,"5min":4.0570000000000004,"15min":4.0570000000000004},"last":1.464}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":10,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":1.732,"15min":1.732},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"last":1.528},{"interface":"front","average":{"1min":1.752,"5min":1.752,"15min":1.752},"min":{"1min":1.153,"5min":1.153,"15min":1.153},"max":{"1min":4.2489999999999997,"5min":4.2489999999999997,"15min":4.2489999999999997},"last":1.2110000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7350000000000001}]},{"osd":15,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"min":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":1.9379999999999999},{"interface":"front","average":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"min":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"max":{"1min":3.8769999999999998,"5min":3.8769999999999998,"15min":3.8769999999999998},"last":1.6639999999999999}]}]},{"osd":10,"up_from":23,"seq":98784247828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1069328,"kb_used_data":20744,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93302512,"statfs":{"total":96636764160,"available":95541772288,"internally_reserved":1073741824,"allocated":21241856,"data_stored":31679191,"data_compressed":141646,"data_compressed_allocated":294912,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":0.752}]},{"osd":1,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.254},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.78100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":3,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.341}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":7,"last update":"Mon Mar 9 00:25:45 2020","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":2.1459999999999999,"5min":2.1459999999999999,"15min":2.1459999999999999},"last":1.159},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.2310000000000001}]},{"osd":8,"last update":"Mon Mar 9 00:25:45 2020","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.999,"5min":0.999,"15min":0.999},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.04},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.369}]},{"osd":9,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.607,"15min":1.607},"min":{"1min":1.167,"5min":1.167,"15min":1.167},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.1869999999999998},"last":1.452},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.3180000000000001}]},{"osd":11,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"max":{"1min":2.214,"5min":2.214,"15min":2.214},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":1.278,"5min":1.278,"15min":1.278},"max":{"1min":2.6070000000000002,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.502}]},{"osd":12,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"min":{"1min":1.496,"5min":1.496,"15min":1.496},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.4890000000000001}]},{"osd":13,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"min":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"min":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.6459999999999999}]},{"osd":14,"last update":"Mon Mar 9 00:25:22 2020","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.768,"15min":1.768},"min":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.778},{"interface":"front","average":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"min":{"1min":1.534,"5min":1.534,"15min":1.534},"max":{"1min":3.2989999999999999,"5min":3.2989999999999999,"15min":3.2989999999999999},"last":1.6040000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.919}]}]},{"osd":3,"up_from":24,"seq":103079215124,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1055552,"kb_used_data":6968,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93316288,"statfs":{"total":96636764160,"available":95555878912,"internally_reserved":1073741824,"allocated":7135232,"data_stored":9788211,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:30 2020","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":4.0979999999999999,"5min":4.0979999999999999,"15min":4.0979999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":5.6319999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"last":1.256}]},{"osd":1,"last update":"Mon Mar 9 00:25:45 2020","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":1.5720000000000001}]},{"osd":2,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":5.6059999999999999,"5min":5.6059999999999999,"15min":5.6059999999999999},"last":1.034},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":6.6109999999999998,"5min":6.6109999999999998,"15min":6.6109999999999998},"last":1.0780000000000001}]},{"osd":4,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0369999999999999,"5min":3.0369999999999999,"15min":3.0369999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":4.5670000000000002,"5min":4.5670000000000002,"15min":4.5670000000000002},"last":1.3400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":7,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.234,"15min":1.234},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":4.3710000000000004,"5min":4.3710000000000004,"15min":4.3710000000000004},"last":1.1479999999999999}]},{"osd":8,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":7.1070000000000002,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.7450000000000001}]},{"osd":9,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":5.1029999999999998,"5min":5.1029999999999998,"15min":5.1029999999999998},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":4.923,"5min":4.923,"15min":4.923},"last":1.6679999999999999}]},{"osd":10,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.629,"15min":1.629},"min":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":1.7010000000000001}]},{"osd":11,"last update":"Mon Mar 9 00:25:23 2020","interfaces":[{"interface":"back","average":{"1min":1.994,"5min":1.994,"15min":1.994},"min":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"max":{"1min":8.4990000000000006,"5min":8.4990000000000006,"15min":8.4990000000000006},"last":1.7709999999999999},{"interface":"front","average":{"1min":1.895,"5min":1.895,"15min":1.895},"min":{"1min":1.383,"5min":1.383,"15min":1.383},"max":{"1min":4.633,"5min":4.633,"15min":4.633},"last":2.1280000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5259999999999998}]},{"osd":15,"last update":"Mon Mar 9 00:25:45 2020","interfaces":[{"interface":"back","average":{"1min":2.109,"5min":2.109,"15min":2.109},"min":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"max":{"1min":4.9660000000000002,"5min":4.9660000000000002,"15min":4.9660000000000002},"last":2.8530000000000002},{"interface":"front","average":{"1min":1.75,"5min":1.75,"15min":1.75},"min":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":1.903}]}]},{"osd":1,"up_from":24,"seq":103079215123,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1070268,"kb_used_data":21684,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301572,"statfs":{"total":96636764160,"available":95540809728,"internally_reserved":1073741824,"allocated":22204416,"data_stored":35773235,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":1.3620000000000001}]},{"osd":2,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.93000000000000005}]},{"osd":3,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":1.119},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":0.96199999999999997}]},{"osd":4,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"min":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.661},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":1.171}]},{"osd":5,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.105,"5min":1.105,"15min":1.105},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.4970000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":7,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":1.8320000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"min":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.3340000000000001}]},{"osd":8,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"min":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.915}]},{"osd":9,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.8580000000000001},"min":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"max":{"1min":2.5760000000000001,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"min":{"1min":1.52,"5min":1.52,"15min":1.52},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":2.069}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.615}]},{"osd":14,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.7490000000000001},"min":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":1.776},{"interface":"front","average":{"1min":1.917,"5min":1.917,"15min":1.917},"min":{"1min":1.47,"5min":1.47,"15min":1.47},"max":{"1min":2.4460000000000002,"5min":2.4460000000000002,"15min":2.4460000000000002},"last":1.806}]}]},{"osd":2,"up_from":24,"seq":103079215123,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1061304,"kb_used_data":12720,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310536,"statfs":{"total":96636764160,"available":95549988864,"internally_reserved":1073741824,"allocated":13025280,"data_stored":16224983,"data_compressed":141646,"data_compressed_allocated":294912,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.78100000000000003}]},{"osd":1,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.157}]},{"osd":3,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.8759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":0.873},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.79900000000000004}]},{"osd":4,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":2.3740000000000001},"last":1.038},{"interface":"front","average":{"1min":1.133,"5min":1.133,"15min":1.133},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":1.0069999999999999}]},{"osd":5,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.452,"15min":1.452},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.2410000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.204},{"interface":"front","average":{"1min":1.571,"5min":1.571,"15min":1.571},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":1.137}]},{"osd":8,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.3049999999999999}]},{"osd":9,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.3370000000000002},"last":1.5329999999999999}]},{"osd":10,"last update":"Mon Mar 9 00:25:21 2020","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.722,"15min":1.722},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"min":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"max":{"1min":3.3940000000000001,"5min":3.3940000000000001,"15min":3.3940000000000001},"last":1.4430000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]}]},{"osd":4,"up_from":24,"seq":103079215124,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1073348,"kb_used_data":24764,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93298492,"statfs":{"total":96636764160,"available":95537655808,"internally_reserved":1073741824,"allocated":25358336,"data_stored":35416883,"data_compressed":143641,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,5,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.4429999999999996}]},{"osd":1,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":4.9139999999999997},"last":4.6289999999999996},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.984,"5min":2.984,"15min":2.984},"last":5.6369999999999996}]},{"osd":3,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":5.0730000000000004,"5min":5.0730000000000004,"15min":5.0730000000000004},"last":4.7610000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":3.4390000000000001}]},{"osd":5,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":6.5540000000000003},{"interface":"front","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":6.0439999999999996,"5min":6.0439999999999996,"15min":6.0439999999999996},"last":6.0540000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.9210000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]},{"osd":10,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":3.956,"5min":3.956,"15min":3.956},"last":5.9050000000000002},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":3.7010000000000001,"5min":3.7010000000000001,"15min":3.7010000000000001},"last":3.7130000000000001}]},{"osd":11,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.821,"15min":1.821},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":6.2240000000000002,"5min":6.2240000000000002,"15min":6.2240000000000002},"last":6.2510000000000003},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":3.581}]},{"osd":12,"last update":"Mon Mar 9 00:25:25 2020","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":5.0309999999999997},"last":3.9750000000000001},{"interface":"front","average":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"min":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"max":{"1min":4.0629999999999997,"5min":4.0629999999999997,"15min":4.0629999999999997},"last":9.5139999999999993}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1120000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.391}]}]}]}} 2020-03-09T00:25:49.683 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2020-03-09T00:25:49.684 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2020-03-09T00:25:49.769 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:49.767+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:49.790 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:49.791 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:49.791 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:49.791 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:49.791 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.791 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:196 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~540672] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:49.791 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 61.962278435000002, 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501, 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.792 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 196 2020-03-09T00:25:49.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.793 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.793 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.793 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.793 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:49.793 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.793 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.794 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.825692+0000", 2020-03-09T00:25:49.794 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035415000000000002 2020-03-09T00:25:49.794 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.794 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.794 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826046+0000", 2020-03-09T00:25:49.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.2260000000000005e-06 2020-03-09T00:25:49.795 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.795 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.795 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.795 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826054+0000", 2020-03-09T00:25:49.795 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3873000000000001e-05 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826068+0000", 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0900000000000001e-06 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.796 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826074+0000", 2020-03-09T00:25:49.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0126e-05 2020-03-09T00:25:49.797 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.797 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.797 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.797 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826134+0000", 2020-03-09T00:25:49.797 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0001117 2020-03-09T00:25:49.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.798 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.798 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.826246+0000", 2020-03-09T00:25:49.798 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00124165 2020-03-09T00:25:49.798 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.798 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827488+0000", 2020-03-09T00:25:49.799 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088469059999999999 2020-03-09T00:25:49.799 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.799 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.799 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.799 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836335+0000", 2020-03-09T00:25:49.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00035733999999999999 2020-03-09T00:25:49.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.800 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.800 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.800 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836692+0000", 2020-03-09T00:25:49.800 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1440000000000005e-05 2020-03-09T00:25:49.800 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.801 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.801 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.801 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836783+0000", 2020-03-09T00:25:49.801 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011091501 2020-03-09T00:25:49.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.801 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.801 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.802 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.802 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.802 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:199 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:49.802 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:49.802 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 61.956532316000001, 2020-03-09T00:25:49.802 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149, 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 199 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.803 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.804 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.804 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:49.804 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.804 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.804 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.805 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831438+0000", 2020-03-09T00:25:49.805 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1242999999999998e-05 2020-03-09T00:25:49.805 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.805 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.805 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831479+0000", 2020-03-09T00:25:49.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5740000000000001e-06 2020-03-09T00:25:49.806 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.806 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.806 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.806 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831486+0000", 2020-03-09T00:25:49.806 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1513e-05 2020-03-09T00:25:49.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.807 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.807 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.807 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831497+0000", 2020-03-09T00:25:49.807 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2760000000000002e-06 2020-03-09T00:25:49.807 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.831502+0000", 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053448499999999999 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.808 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836846+0000", 2020-03-09T00:25:49.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000103844 2020-03-09T00:25:49.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.809 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.809 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:49.809 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.836950+0000", 2020-03-09T00:25:49.809 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045940299999999998 2020-03-09T00:25:49.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.810 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841544+0000", 2020-03-09T00:25:49.810 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105018 2020-03-09T00:25:49.810 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.810 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.810 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.841649+0000", 2020-03-09T00:25:49.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002376801 2020-03-09T00:25:49.811 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.811 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.811 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.811 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.844026+0000", 2020-03-09T00:25:49.811 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012588149 2020-03-09T00:25:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:346 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:49.812 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:49.813 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.962438718999998, 2020-03-09T00:25:49.813 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654, 2020-03-09T00:25:49.813 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.813 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.813 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.814 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.814 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 346 2020-03-09T00:25:49.814 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.814 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.814 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.814 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.814 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825532+0000", 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014902900000000001 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.815 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825681+0000", 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825687+0000", 2020-03-09T00:25:49.816 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6099999999999995e-06 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825696+0000", 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6149999999999999e-06 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.817 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825701+0000", 2020-03-09T00:25:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4975000000000002e-05 2020-03-09T00:25:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825746+0000", 2020-03-09T00:25:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0597e-05 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.825836+0000", 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00091597100000000002 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.826752+0000", 2020-03-09T00:25:49.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046291539999999999 2020-03-09T00:25:49.820 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.820 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.820 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.820 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831382+0000", 2020-03-09T00:25:49.820 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098416760000000006 2020-03-09T00:25:49.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.821 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.821 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841223+0000", 2020-03-09T00:25:49.821 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000100926 2020-03-09T00:25:49.821 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.821 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.822 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.822 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841324+0000", 2020-03-09T00:25:49.822 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015792654 2020-03-09T00:25:49.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.822 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.822 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.822 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:347 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~622592] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.955170897000002, 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999, 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.823 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 347 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.824 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:49.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.825 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.825 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.825 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.825 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832799+0000", 2020-03-09T00:25:49.825 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000469071 2020-03-09T00:25:49.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.826 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.826 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833268+0000", 2020-03-09T00:25:49.826 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7889999999999995e-06 2020-03-09T00:25:49.826 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.826 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833276+0000", 2020-03-09T00:25:49.827 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6033999999999999e-05 2020-03-09T00:25:49.827 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.827 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.827 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.827 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833292+0000", 2020-03-09T00:25:49.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8850000000000001e-06 2020-03-09T00:25:49.828 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.828 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.828 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.828 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833299+0000", 2020-03-09T00:25:49.828 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833343+0000", 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000105574 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.829 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833448+0000", 2020-03-09T00:25:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00094406100000000005 2020-03-09T00:25:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834392+0000", 2020-03-09T00:25:49.830 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088904150000000005 2020-03-09T00:25:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843283+0000", 2020-03-09T00:25:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052783500000000004 2020-03-09T00:25:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.831 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843811+0000", 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1547999999999998e-05 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843902+0000", 2020-03-09T00:25:49.832 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011102829999999999 2020-03-09T00:25:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:348 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~147456] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:49.833 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.954614806999999, 2020-03-09T00:25:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999, 2020-03-09T00:25:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.834 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 348 2020-03-09T00:25:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.835 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:49.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.836 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.836 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.836 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833355+0000", 2020-03-09T00:25:49.836 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019096900000000001 2020-03-09T00:25:49.836 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833546+0000", 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4060000000000002e-06 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.837 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833553+0000", 2020-03-09T00:25:49.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1321000000000001e-05 2020-03-09T00:25:49.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.838 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.838 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.838 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:49.838 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6310000000000002e-06 2020-03-09T00:25:49.838 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.839 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833569+0000", 2020-03-09T00:25:49.839 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037440260000000001 2020-03-09T00:25:49.839 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.839 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.839 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837313+0000", 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134988 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.837448+0000", 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001155454 2020-03-09T00:25:49.840 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.841 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.838603+0000", 2020-03-09T00:25:49.841 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046829050000000002 2020-03-09T00:25:49.841 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.841 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.841 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843286+0000", 2020-03-09T00:25:49.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011927330000000001 2020-03-09T00:25:49.842 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.842 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.842 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.842 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844479+0000", 2020-03-09T00:25:49.842 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5680000000000005e-05 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.844575+0000", 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011219112999999999 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.843 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.844 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.844 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.844 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:349 3.5s0 3:ada90060:::smithi10428818-8:head [setxattr _header (56) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:49.844 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:49.844 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.954349247000003, 2020-03-09T00:25:49.844 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001, 2020-03-09T00:25:49.844 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.845 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.845 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.845 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.845 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.845 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 349 2020-03-09T00:25:49.845 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.845 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.846 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:49.846 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.846 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.846 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833621+0000", 2020-03-09T00:25:49.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.8177999999999997e-05 2020-03-09T00:25:49.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.847 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.847 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.847 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833689+0000", 2020-03-09T00:25:49.847 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3619999999999997e-06 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833695+0000", 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1154e-05 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.848 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833706+0000", 2020-03-09T00:25:49.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5030000000000002e-06 2020-03-09T00:25:49.849 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.849 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.849 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.849 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833712+0000", 2020-03-09T00:25:49.849 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0079814919999999998 2020-03-09T00:25:49.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.850 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.850 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841694+0000", 2020-03-09T00:25:49.850 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013282600000000001 2020-03-09T00:25:49.850 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.850 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.841826+0000", 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071321600000000002 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.842540+0000", 2020-03-09T00:25:49.851 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00074733700000000004 2020-03-09T00:25:49.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.852 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.852 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843287+0000", 2020-03-09T00:25:49.852 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001845368 2020-03-09T00:25:49.852 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.852 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845132+0000", 2020-03-09T00:25:49.853 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6600000000000004e-05 2020-03-09T00:25:49.853 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.853 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.853 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.853 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845219+0000", 2020-03-09T00:25:49.854 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011598036000000001 2020-03-09T00:25:49.854 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.854 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.854 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.854 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.854 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.854 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:350 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:49.855 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:49.855 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 46.954219590000001, 2020-03-09T00:25:49.855 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417, 2020-03-09T00:25:49.855 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.855 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:49.855 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.855 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 350 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:49.856 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.857 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.857 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.857 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833751+0000", 2020-03-09T00:25:49.857 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0074999999999998e-05 2020-03-09T00:25:49.857 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.858 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833791+0000", 2020-03-09T00:25:49.858 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.1530000000000003e-06 2020-03-09T00:25:49.858 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.858 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.858 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833800+0000", 2020-03-09T00:25:49.859 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3621000000000001e-05 2020-03-09T00:25:49.859 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.859 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.859 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.859 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833813+0000", 2020-03-09T00:25:49.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.9899999999999999e-06 2020-03-09T00:25:49.860 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.860 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.860 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.860 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.833817+0000", 2020-03-09T00:25:49.860 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0093306459999999997 2020-03-09T00:25:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843148+0000", 2020-03-09T00:25:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012790100000000001 2020-03-09T00:25:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.861 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:49.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.843276+0000", 2020-03-09T00:25:49.862 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025486789999999999 2020-03-09T00:25:49.862 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.862 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.862 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.862 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845825+0000", 2020-03-09T00:25:49.863 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120695 2020-03-09T00:25:49.863 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.863 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.863 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845945+0000", 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0033866569999999999 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.849332+0000", 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015581417 2020-03-09T00:25:49.864 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.865 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.865 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.865 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.865 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.865 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.8s5 59/54 ECSubWrite(tid=6, reqid=client.4751.0:486, at_version=59'27, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:49.865 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:49.865 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 40.969078787999997, 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012816759, 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:49.866 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818891+0000", 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000102045 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.867 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.818993+0000", 2020-03-09T00:25:49.868 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2311999999999999e-05 2020-03-09T00:25:49.868 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.868 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.868 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.868 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819016+0000", 2020-03-09T00:25:49.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6019999999999999e-05 2020-03-09T00:25:49.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.869 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.869 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.869 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819032+0000", 2020-03-09T00:25:49.869 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4910000000000002e-06 2020-03-09T00:25:49.869 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.870 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.870 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.819038+0000", 2020-03-09T00:25:49.870 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0023069140000000002 2020-03-09T00:25:49.870 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.870 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.871 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.871 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821345+0000", 2020-03-09T00:25:49.871 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3864e-05 2020-03-09T00:25:49.871 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.871 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.821359+0000", 2020-03-09T00:25:49.872 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010305182 2020-03-09T00:25:49.872 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.872 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.872 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.872 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831664+0000", 2020-03-09T00:25:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3930999999999997e-05 2020-03-09T00:25:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.831708+0000", 2020-03-09T00:25:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012816759 2020-03-09T00:25:49.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.874 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.874 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.874 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.874 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.874 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 59/32 ECSubWrite(tid=162, reqid=osd.12.0:162, at_version=59'79, trim_to=0'0, roll_forward_to=58'75))", 2020-03-09T00:25:49.875 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:49.875 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 40.96237283, 2020-03-09T00:25:49.875 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013484734, 2020-03-09T00:25:49.875 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.875 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:49.875 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.875 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.876 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.876 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:49.876 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.876 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.876 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825597+0000", 2020-03-09T00:25:49.877 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000120926 2020-03-09T00:25:49.877 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.877 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.877 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.877 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825718+0000", 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8563000000000003e-05 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825757+0000", 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3974999999999999e-05 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.878 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.879 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825791+0000", 2020-03-09T00:25:49.879 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2510999999999999e-05 2020-03-09T00:25:49.879 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.879 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.879 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.825803+0000", 2020-03-09T00:25:49.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062712599999999999 2020-03-09T00:25:49.880 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.880 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.880 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.880 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826430+0000", 2020-03-09T00:25:49.880 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2381000000000001e-05 2020-03-09T00:25:49.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.881 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.881 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.881 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.826443+0000", 2020-03-09T00:25:49.881 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012584521 2020-03-09T00:25:49.881 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.882 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839027+0000", 2020-03-09T00:25:49.882 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4731000000000002e-05 2020-03-09T00:25:49.882 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.882 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.882 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.883 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.839082+0000", 2020-03-09T00:25:49.883 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013484734 2020-03-09T00:25:49.883 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.883 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.884 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.884 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.884 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:650 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~3244032,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e76)", 2020-03-09T00:25:49.885 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:25.784887+0000", 2020-03-09T00:25:49.885 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.003083039, 2020-03-09T00:25:49.885 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017635736999999999, 2020-03-09T00:25:49.885 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.885 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:49.885 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.885 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.886 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.886 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 650 2020-03-09T00:25:49.886 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.886 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.886 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.886 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.886 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784887+0000", 2020-03-09T00:25:49.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.887 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.887 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.887 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784887+0000", 2020-03-09T00:25:49.887 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5030999999999995e-05 2020-03-09T00:25:49.887 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.888 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784952+0000", 2020-03-09T00:25:49.888 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4569999999999999e-06 2020-03-09T00:25:49.888 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.888 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.888 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784960+0000", 2020-03-09T00:25:49.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4148000000000001e-05 2020-03-09T00:25:49.889 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.889 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.889 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.889 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784974+0000", 2020-03-09T00:25:49.889 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2119999999999997e-06 2020-03-09T00:25:49.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.890 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.890 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.784981+0000", 2020-03-09T00:25:49.890 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4878e-05 2020-03-09T00:25:49.890 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.890 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785026+0000", 2020-03-09T00:25:49.891 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.5020000000000003e-06 2020-03-09T00:25:49.891 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.891 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.891 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:49.891 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785033+0000", 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002150463 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787184+0000", 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000112349 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.892 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787296+0000", 2020-03-09T00:25:49.893 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.015226697000000001 2020-03-09T00:25:49.893 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.893 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.893 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.893 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.802523+0000", 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017635736999999999 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:651 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~3244032,checksum] snapc 0=[] ondisk+read+known_if_redirected e76)", 2020-03-09T00:25:49.894 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:25.785035+0000", 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.002935164, 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020421043999999999, 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 651 2020-03-09T00:25:49.895 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785035+0000", 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.896 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785035+0000", 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5442000000000001e-05 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785101+0000", 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5069999999999997e-06 2020-03-09T00:25:49.897 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.898 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785107+0000", 2020-03-09T00:25:49.898 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0242e-05 2020-03-09T00:25:49.898 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.898 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.898 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785117+0000", 2020-03-09T00:25:49.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7919999999999999e-06 2020-03-09T00:25:49.899 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.899 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.899 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.899 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785121+0000", 2020-03-09T00:25:49.899 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9161000000000001e-05 2020-03-09T00:25:49.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.900 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.900 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785150+0000", 2020-03-09T00:25:49.900 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6180000000000003e-06 2020-03-09T00:25:49.900 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.900 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785156+0000", 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0025586839999999999 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787715+0000", 2020-03-09T00:25:49.901 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011526799999999999 2020-03-09T00:25:49.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.902 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.902 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.787830+0000", 2020-03-09T00:25:49.902 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017626329999999999 2020-03-09T00:25:49.902 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.902 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.903 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.903 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.805456+0000", 2020-03-09T00:25:49.903 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020421043999999999 2020-03-09T00:25:49.903 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.903 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.903 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:652 3.5s0 3:ada90060:::smithi10428818-8:head [read 0~3244032,checksum] snapc 0=[] ondisk+read+known_if_redirected e76)", 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:25.785168+0000", 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.002802363000001, 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023422449000000001, 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.904 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:49.905 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.905 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.905 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.905 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 652 2020-03-09T00:25:49.905 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.905 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.906 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.906 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785168+0000", 2020-03-09T00:25:49.906 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.906 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.906 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785168+0000", 2020-03-09T00:25:49.907 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1877000000000004e-05 2020-03-09T00:25:49.907 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.907 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.907 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.907 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785230+0000", 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0100000000000003e-06 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785235+0000", 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5959999999999993e-06 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.908 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785244+0000", 2020-03-09T00:25:49.909 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8240000000000001e-06 2020-03-09T00:25:49.909 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.909 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.909 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.909 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785248+0000", 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1299e-05 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785279+0000", 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.6649999999999998e-06 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.910 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:49.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.785289+0000", 2020-03-09T00:25:49.911 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029630780000000001 2020-03-09T00:25:49.911 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.911 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.911 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.911 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.788252+0000", 2020-03-09T00:25:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012021199999999999 2020-03-09T00:25:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.788372+0000", 2020-03-09T00:25:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.020217888 2020-03-09T00:25:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.913 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.913 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.913 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:25.808590+0000", 2020-03-09T00:25:49.913 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.023422449000000001 2020-03-09T00:25:49.913 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.913 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.913 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.914 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.914 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.914 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.9s4 81/67 ECSubWrite(tid=263, reqid=client.4751.0:687, at_version=81'142, trim_to=0'0, roll_forward_to=71'139))", 2020-03-09T00:25:49.914 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:30.829030+0000", 2020-03-09T00:25:49.914 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 18.958940291000001, 2020-03-09T00:25:49.914 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01118852, 2020-03-09T00:25:49.915 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.915 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:49.915 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.915 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.915 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.915 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.829030+0000", 2020-03-09T00:25:49.915 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.916 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.916 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.829030+0000", 2020-03-09T00:25:49.916 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013312039999999999 2020-03-09T00:25:49.916 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.916 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830361+0000", 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2396e-05 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830384+0000", 2020-03-09T00:25:49.917 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1953999999999999e-05 2020-03-09T00:25:49.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.918 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.918 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830395+0000", 2020-03-09T00:25:49.918 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4450000000000004e-06 2020-03-09T00:25:49.918 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.918 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830401+0000", 2020-03-09T00:25:49.919 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.371e-05 2020-03-09T00:25:49.919 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.919 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.919 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.919 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830445+0000", 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0094e-05 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.830455+0000", 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097253349999999999 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.920 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.840180+0000", 2020-03-09T00:25:49.921 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8381999999999999e-05 2020-03-09T00:25:49.921 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.921 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.921 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.921 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.840218+0000", 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01118852 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.8s5 86/67 ECSubWrite(tid=46, reqid=client.4751.0:772, at_version=86'54, trim_to=0'0, roll_forward_to=85'52))", 2020-03-09T00:25:49.922 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:35.850112+0000", 2020-03-09T00:25:49.923 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 13.937858599, 2020-03-09T00:25:49.923 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012875307000000001, 2020-03-09T00:25:49.923 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.923 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:49.923 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.923 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.923 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.850112+0000", 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.850112+0000", 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001778389 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.924 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.851890+0000", 2020-03-09T00:25:49.925 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.9169000000000001e-05 2020-03-09T00:25:49.925 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.925 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.925 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.925 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.851919+0000", 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0966e-05 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.851930+0000", 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.1259999999999997e-06 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.926 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.851935+0000", 2020-03-09T00:25:49.927 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1736999999999998e-05 2020-03-09T00:25:49.927 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.927 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.927 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.927 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.851967+0000", 2020-03-09T00:25:49.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.396e-06 2020-03-09T00:25:49.928 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.928 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.928 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.928 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.851974+0000", 2020-03-09T00:25:49.928 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010973524 2020-03-09T00:25:49.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.929 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.929 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.862948+0000", 2020-03-09T00:25:49.929 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8999999999999999e-05 2020-03-09T00:25:49.929 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.929 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.930 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.930 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:35.862987+0000", 2020-03-09T00:25:49.930 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.012875307000000001 2020-03-09T00:25:49.930 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.930 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.930 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.930 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.931 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.931 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as5 94/32 ECSubWrite(tid=40, reqid=client.4751.0:909, at_version=94'38, trim_to=0'0, roll_forward_to=84'36))", 2020-03-09T00:25:49.931 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.882536+0000", 2020-03-09T00:25:49.931 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.9054343290000002, 2020-03-09T00:25:49.931 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011492485, 2020-03-09T00:25:49.931 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.931 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:49.932 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.932 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.932 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882536+0000", 2020-03-09T00:25:49.932 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.932 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.932 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882536+0000", 2020-03-09T00:25:49.933 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012398800000000001 2020-03-09T00:25:49.933 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.933 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.933 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.933 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882660+0000", 2020-03-09T00:25:49.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6817000000000001e-05 2020-03-09T00:25:49.934 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.934 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.934 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.934 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882687+0000", 2020-03-09T00:25:49.934 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7538e-05 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882704+0000", 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3300000000000001e-06 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.935 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882712+0000", 2020-03-09T00:25:49.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0037651709999999999 2020-03-09T00:25:49.936 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.936 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.936 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.936 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.886477+0000", 2020-03-09T00:25:49.936 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5943999999999998e-05 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.886493+0000", 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074775850000000001 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.937 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.893970+0000", 2020-03-09T00:25:49.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8112000000000002e-05 2020-03-09T00:25:49.938 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.938 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.938 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.938 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894028+0000", 2020-03-09T00:25:49.938 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011492485 2020-03-09T00:25:49.939 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.939 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.939 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.939 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.939 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.939 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:920 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~0] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:49.939 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.950706+0000", 2020-03-09T00:25:49.940 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8372638200000004, 2020-03-09T00:25:49.940 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010993134, 2020-03-09T00:25:49.940 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.940 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.940 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.940 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.940 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.941 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 920 2020-03-09T00:25:49.941 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.941 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.941 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.941 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.941 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950706+0000", 2020-03-09T00:25:49.941 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950706+0000", 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2985000000000002e-05 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.942 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950759+0000", 2020-03-09T00:25:49.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8449999999999992e-06 2020-03-09T00:25:49.943 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.943 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.943 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.943 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950768+0000", 2020-03-09T00:25:49.943 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3633e-05 2020-03-09T00:25:49.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.944 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.944 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950782+0000", 2020-03-09T00:25:49.944 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4420000000000004e-06 2020-03-09T00:25:49.944 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.944 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950791+0000", 2020-03-09T00:25:49.945 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4393000000000004e-05 2020-03-09T00:25:49.945 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.945 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.945 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.945 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950866+0000", 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014953899999999999 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951015+0000", 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0010209500000000001 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.946 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952036+0000", 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088870540000000001 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.960923+0000", 2020-03-09T00:25:49.947 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00046016699999999999 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.961383+0000", 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00031612600000000002 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.948 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.949 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.961700+0000", 2020-03-09T00:25:49.949 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010993134 2020-03-09T00:25:49.949 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.949 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.949 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.949 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:921 3.5s0 3:ada90060:::smithi10428818-8:head [rollback 38] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.950850+0000", 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8371197720000003, 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013313148, 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.950 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.951 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.951 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 921 2020-03-09T00:25:49.951 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.951 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.951 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.951 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950850+0000", 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950850+0000", 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4200999999999999e-05 2020-03-09T00:25:49.952 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950895+0000", 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0109999999999997e-06 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.953 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950901+0000", 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0886999999999999e-05 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950912+0000", 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7099999999999998e-06 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.954 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950916+0000", 2020-03-09T00:25:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034554049999999999 2020-03-09T00:25:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.954372+0000", 2020-03-09T00:25:49.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000123776 2020-03-09T00:25:49.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.956 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.956 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.956 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.954495+0000", 2020-03-09T00:25:49.956 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00078748299999999995 2020-03-09T00:25:49.956 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.955283+0000", 2020-03-09T00:25:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056838590000000003 2020-03-09T00:25:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.957 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.960967+0000", 2020-03-09T00:25:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0029006589999999999 2020-03-09T00:25:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.963867+0000", 2020-03-09T00:25:49.958 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00029615700000000002 2020-03-09T00:25:49.959 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.959 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.959 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.959 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.964164+0000", 2020-03-09T00:25:49.959 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013313148 2020-03-09T00:25:49.959 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.959 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.960 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.960 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.960 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.960 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:922 3.5s0 3:ada90060:::smithi10428818-8:head [append 0~0] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:49.960 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.950969+0000", 2020-03-09T00:25:49.960 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.837000808, 2020-03-09T00:25:49.961 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014951986, 2020-03-09T00:25:49.961 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.961 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:49.961 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.961 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.961 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.961 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 922 2020-03-09T00:25:49.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.962 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.962 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.962 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.962 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950969+0000", 2020-03-09T00:25:49.962 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.962 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.963 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.963 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.963 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.950969+0000", 2020-03-09T00:25:49.963 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4478999999999999e-05 2020-03-09T00:25:49.963 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.964 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.964 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951014+0000", 2020-03-09T00:25:49.964 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9970000000000004e-06 2020-03-09T00:25:49.964 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.964 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.964 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951020+0000", 2020-03-09T00:25:49.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1610999999999999e-05 2020-03-09T00:25:49.965 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.965 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.965 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.965 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951032+0000", 2020-03-09T00:25:49.965 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4599999999999996e-06 2020-03-09T00:25:49.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.966 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.966 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951036+0000", 2020-03-09T00:25:49.966 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0086648939999999994 2020-03-09T00:25:49.966 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.966 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.959701+0000", 2020-03-09T00:25:49.967 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016050099999999999 2020-03-09T00:25:49.967 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.967 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.967 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.967 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.959861+0000", 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00082145199999999997 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.960683+0000", 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0014648529999999999 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.968 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.969 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:49.969 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.962148+0000", 2020-03-09T00:25:49.969 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034852609999999999 2020-03-09T00:25:49.969 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.969 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:49.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.965633+0000", 2020-03-09T00:25:49.970 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000288478 2020-03-09T00:25:49.970 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.970 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.970 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.970 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.965921+0000", 2020-03-09T00:25:49.971 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014951986 2020-03-09T00:25:49.971 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.971 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.971 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.971 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.971 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.971 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:929 3.5s0 3:a0e51e4d:::smithi10428818-41:head [copy-from ver 0] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:49.972 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.951108+0000", 2020-03-09T00:25:49.972 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8368625940000003, 2020-03-09T00:25:49.972 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016577794999999999, 2020-03-09T00:25:49.972 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.972 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:49.973 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.973 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.973 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.973 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 929 2020-03-09T00:25:49.973 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.973 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.973 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951108+0000", 2020-03-09T00:25:49.974 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.974 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.974 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.974 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.974 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951108+0000", 2020-03-09T00:25:49.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.1732999999999999e-05 2020-03-09T00:25:49.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.975 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.975 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.975 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951179+0000", 2020-03-09T00:25:49.975 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9870000000000002e-06 2020-03-09T00:25:49.975 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.976 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951186+0000", 2020-03-09T00:25:49.976 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1235e-05 2020-03-09T00:25:49.976 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.976 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.976 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951198+0000", 2020-03-09T00:25:49.977 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8570000000000003e-06 2020-03-09T00:25:49.977 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.977 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.977 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.977 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951202+0000", 2020-03-09T00:25:49.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010630753999999999 2020-03-09T00:25:49.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.978 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.978 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.978 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.961833+0000", 2020-03-09T00:25:49.978 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013326500000000001 2020-03-09T00:25:49.978 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.979 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.961966+0000", 2020-03-09T00:25:49.979 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005718964 2020-03-09T00:25:49.979 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.979 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.979 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.980 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.967685+0000", 2020-03-09T00:25:49.980 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016577794999999999 2020-03-09T00:25:49.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.980 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.980 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.980 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.980 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.981 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:930 3.5s0 3:a0e51e4d:::smithi10428818-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e94)", 2020-03-09T00:25:49.981 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.951246+0000", 2020-03-09T00:25:49.981 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 5.8367241759999997, 2020-03-09T00:25:49.981 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017217171, 2020-03-09T00:25:49.981 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:49.981 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:49.981 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:49.982 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:49.982 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:49.982 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 930 2020-03-09T00:25:49.982 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.982 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:49.982 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:49.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951246+0000", 2020-03-09T00:25:49.983 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:49.983 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.983 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.983 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:49.983 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951246+0000", 2020-03-09T00:25:49.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4733e-05 2020-03-09T00:25:49.984 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.984 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.984 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:49.984 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951291+0000", 2020-03-09T00:25:49.984 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4289e-05 2020-03-09T00:25:49.985 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.985 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.986 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:49.986 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951305+0000", 2020-03-09T00:25:49.986 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4997999999999999e-05 2020-03-09T00:25:49.986 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.986 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951320+0000", 2020-03-09T00:25:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.0430000000000001e-06 2020-03-09T00:25:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.951324+0000", 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010862037 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.962186+0000", 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9579000000000006e-05 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:49.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.962256+0000", 2020-03-09T00:25:49.989 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056700830000000002 2020-03-09T00:25:49.989 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.989 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.989 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:49.989 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.967926+0000", 2020-03-09T00:25:49.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000128355 2020-03-09T00:25:49.990 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.990 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.990 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:49.990 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.968054+0000", 2020-03-09T00:25:49.990 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000409054 2020-03-09T00:25:49.991 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:49.991 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:49.991 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:49.991 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.968463+0000", 2020-03-09T00:25:49.991 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.017217171 2020-03-09T00:25:49.991 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.991 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.992 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.992 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:49.992 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:49.992 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:49.994 INFO:tasks.ceph.osd.5.smithi050.stderr:2020-03-09T00:25:49.992+0000 7fbc6a949700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27621) UID: 0 2020-03-09T00:25:49.994 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2020-03-09T00:25:50.095 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:50.093+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:50.121 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:50.121 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:50.122 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:50.122 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:50.130 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2020-03-09T00:25:50.196 INFO:tasks.ceph.osd.3.smithi104.stderr:2020-03-09T00:25:50.194+0000 7efda9985700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28003) UID: 0 2020-03-09T00:25:50.246 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:50.246 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:50.246 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:50.246 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:50.246 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:50.254 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2020-03-09T00:25:50.298 INFO:tasks.ceph.osd.9.smithi050.stderr:2020-03-09T00:25:50.296+0000 7f855b794700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 27626) UID: 0 2020-03-09T00:25:50.372 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:50.372 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:50.372 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:50.372 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:50.373 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.373 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:311 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~770048] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:50.373 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:50.373 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 47.585713390000002, 2020-03-09T00:25:50.373 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438, 2020-03-09T00:25:50.374 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.374 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.374 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.374 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.374 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.374 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 311 2020-03-09T00:25:50.374 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.375 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.375 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.375 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:50.375 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.375 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.375 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.783732+0000", 2020-03-09T00:25:50.376 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0014001199999999999 2020-03-09T00:25:50.376 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.376 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.376 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.376 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785132+0000", 2020-03-09T00:25:50.377 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1973999999999999e-05 2020-03-09T00:25:50.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.377 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.377 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.377 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785144+0000", 2020-03-09T00:25:50.377 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1815e-05 2020-03-09T00:25:50.377 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.378 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.378 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785156+0000", 2020-03-09T00:25:50.378 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2449999999999998e-06 2020-03-09T00:25:50.378 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.378 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.378 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785161+0000", 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00063616300000000003 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785797+0000", 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00065134999999999998 2020-03-09T00:25:50.379 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.380 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.380 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.786449+0000", 2020-03-09T00:25:50.380 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00188737 2020-03-09T00:25:50.380 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.380 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.380 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.381 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788336+0000", 2020-03-09T00:25:50.381 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010887625999999999 2020-03-09T00:25:50.381 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.381 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.382 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.382 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799224+0000", 2020-03-09T00:25:50.382 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037060560000000001 2020-03-09T00:25:50.382 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.382 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.802930+0000", 2020-03-09T00:25:50.383 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000460719 2020-03-09T00:25:50.383 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.383 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.383 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.383 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803390+0000", 2020-03-09T00:25:50.384 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019658438 2020-03-09T00:25:50.384 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.384 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.384 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.384 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.384 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.384 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:312 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~376832] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 47.584217242000001, 2020-03-09T00:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001, 2020-03-09T00:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.385 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.386 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.386 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 312 2020-03-09T00:25:50.386 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.386 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.386 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.386 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.386 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785228+0000", 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026595500000000001 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.387 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785494+0000", 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3902e-05 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785508+0000", 2020-03-09T00:25:50.388 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.417e-05 2020-03-09T00:25:50.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.389 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.389 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785522+0000", 2020-03-09T00:25:50.389 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.99e-06 2020-03-09T00:25:50.389 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.389 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.390 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785529+0000", 2020-03-09T00:25:50.390 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083854199999999993 2020-03-09T00:25:50.390 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.390 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.390 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.390 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793915+0000", 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00024809000000000002 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.794163+0000", 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001122184 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.391 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795285+0000", 2020-03-09T00:25:50.392 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.004034358 2020-03-09T00:25:50.392 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.392 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.392 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.392 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799319+0000", 2020-03-09T00:25:50.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042626369999999997 2020-03-09T00:25:50.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.393 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.393 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.393 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.803582+0000", 2020-03-09T00:25:50.393 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00042245199999999997 2020-03-09T00:25:50.393 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.394 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.394 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.394 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804004+0000", 2020-03-09T00:25:50.394 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018776158000000001 2020-03-09T00:25:50.395 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.395 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.395 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.395 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.395 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:313 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] RETRY=1 ondisk+retry+write+known_if_redirected e53)", 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 47.583861681000002, 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001, 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.396 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.397 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.397 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 313 2020-03-09T00:25:50.397 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.397 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.397 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.397 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.397 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785584+0000", 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7169000000000004e-05 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.398 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785681+0000", 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.8849999999999999e-06 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785689+0000", 2020-03-09T00:25:50.399 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5290000000000001e-05 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785704+0000", 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5759999999999996e-06 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.400 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785710+0000", 2020-03-09T00:25:50.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.012205476 2020-03-09T00:25:50.401 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.401 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.401 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.401 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.797915+0000", 2020-03-09T00:25:50.401 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023467400000000001 2020-03-09T00:25:50.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.402 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.402 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.798150+0000", 2020-03-09T00:25:50.402 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00085795900000000004 2020-03-09T00:25:50.402 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.402 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799008+0000", 2020-03-09T00:25:50.403 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00050973499999999998 2020-03-09T00:25:50.403 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.403 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.403 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.403 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799518+0000", 2020-03-09T00:25:50.404 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046306960000000001 2020-03-09T00:25:50.404 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.404 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.404 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.404 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804148+0000", 2020-03-09T00:25:50.405 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00032100500000000002 2020-03-09T00:25:50.405 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.405 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.405 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.406 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.804469+0000", 2020-03-09T00:25:50.406 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.018885465000000001 2020-03-09T00:25:50.406 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.406 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.407 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.408 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.408 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.408 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:314 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:50.408 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:50.409 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 47.583678900999999, 2020-03-09T00:25:50.409 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998, 2020-03-09T00:25:50.409 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.409 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:50.409 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.409 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 314 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:50.410 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.411 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.411 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.411 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785767+0000", 2020-03-09T00:25:50.411 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9129999999999999e-05 2020-03-09T00:25:50.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785816+0000", 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.1280000000000002e-06 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.412 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785823+0000", 2020-03-09T00:25:50.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.24e-05 2020-03-09T00:25:50.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.413 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.413 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.413 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785835+0000", 2020-03-09T00:25:50.413 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1800000000000004e-06 2020-03-09T00:25:50.413 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.414 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.785840+0000", 2020-03-09T00:25:50.414 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014080127 2020-03-09T00:25:50.414 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.414 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.414 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.415 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799920+0000", 2020-03-09T00:25:50.415 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00023000599999999999 2020-03-09T00:25:50.415 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.415 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.415 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:50.415 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.800150+0000", 2020-03-09T00:25:50.415 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0048975440000000002 2020-03-09T00:25:50.416 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.416 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.416 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.416 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805048+0000", 2020-03-09T00:25:50.416 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000221747 2020-03-09T00:25:50.416 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.416 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.417 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.417 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.805270+0000", 2020-03-09T00:25:50.417 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037657239999999998 2020-03-09T00:25:50.417 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.417 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.417 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.417 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.809036+0000", 2020-03-09T00:25:50.418 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.023268985999999998 2020-03-09T00:25:50.418 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.418 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.418 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.418 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.418 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 47.517923862000004, 2020-03-09T00:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001, 2020-03-09T00:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:50.419 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.420 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:50.420 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.420 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.420 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.420 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.421 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.851522+0000", 2020-03-09T00:25:50.421 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0030360970000000002 2020-03-09T00:25:50.421 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.421 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.421 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.421 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854558+0000", 2020-03-09T00:25:50.421 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8904e-05 2020-03-09T00:25:50.422 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.422 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.422 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.422 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854587+0000", 2020-03-09T00:25:50.422 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5423000000000001e-05 2020-03-09T00:25:50.422 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.422 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854602+0000", 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.9139999999999997e-06 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854611+0000", 2020-03-09T00:25:50.423 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2179000000000002e-05 2020-03-09T00:25:50.424 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.424 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.424 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.424 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854673+0000", 2020-03-09T00:25:50.424 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.6330000000000001e-05 2020-03-09T00:25:50.424 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.424 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.854689+0000", 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019248372999999999 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873938+0000", 2020-03-09T00:25:50.425 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3464000000000003e-05 2020-03-09T00:25:50.426 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.426 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.426 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.426 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.873981+0000", 2020-03-09T00:25:50.426 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022459684000000001 2020-03-09T00:25:50.426 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.426 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.427 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.427 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.427 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.427 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:397 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~573440] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:50.427 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:50.427 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.541136549999997, 2020-03-09T00:25:50.427 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99949370299999996, 2020-03-09T00:25:50.428 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.428 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.428 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.428 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.428 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.428 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 397 2020-03-09T00:25:50.428 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.429 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.429 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.429 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:50.429 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.429 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.429 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.430 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.430 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.828309+0000", 2020-03-09T00:25:50.430 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0015862319999999999 2020-03-09T00:25:50.430 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.430 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.430 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.430 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829895+0000", 2020-03-09T00:25:50.431 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.1249999999999999e-06 2020-03-09T00:25:50.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.431 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.431 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.431 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829904+0000", 2020-03-09T00:25:50.431 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5704000000000001e-05 2020-03-09T00:25:50.431 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829920+0000", 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.3140000000000004e-06 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.432 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.829928+0000", 2020-03-09T00:25:50.433 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000122717 2020-03-09T00:25:50.433 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.433 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.433 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.433 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830051+0000", 2020-03-09T00:25:50.433 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9819999999999999e-06 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830056+0000", 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97235390200000005 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.434 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.802410+0000", 2020-03-09T00:25:50.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016568099999999999 2020-03-09T00:25:50.435 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.435 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.435 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.435 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.802576+0000", 2020-03-09T00:25:50.435 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001235402 2020-03-09T00:25:50.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.436 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.436 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.803811+0000", 2020-03-09T00:25:50.436 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013426433999999999 2020-03-09T00:25:50.436 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.436 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.437 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.437 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817237+0000", 2020-03-09T00:25:50.437 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010489160000000001 2020-03-09T00:25:50.437 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.437 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.437 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.437 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827727+0000", 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.6050000000000005e-05 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827803+0000", 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99949370299999996 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.438 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.439 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.439 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.439 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.439 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:398 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~425984] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:50.439 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:50.439 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.538641106, 2020-03-09T00:25:50.439 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99725236100000003, 2020-03-09T00:25:50.440 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.440 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.440 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.440 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.440 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.440 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 398 2020-03-09T00:25:50.440 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.441 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.441 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.441 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:50.441 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.441 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.441 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.830804+0000", 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00051814600000000004 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831322+0000", 2020-03-09T00:25:50.442 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0118e-05 2020-03-09T00:25:50.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.443 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.443 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831333+0000", 2020-03-09T00:25:50.443 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0710999999999999e-05 2020-03-09T00:25:50.443 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.443 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831353+0000", 2020-03-09T00:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.7870000000000004e-06 2020-03-09T00:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.444 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831360+0000", 2020-03-09T00:25:50.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013810100000000001 2020-03-09T00:25:50.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.445 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.445 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.445 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831498+0000", 2020-03-09T00:25:50.445 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3380000000000001e-06 2020-03-09T00:25:50.445 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:50.446 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.831502+0000", 2020-03-09T00:25:50.446 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.97707543200000002 2020-03-09T00:25:50.446 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.446 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.446 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.808577+0000", 2020-03-09T00:25:50.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000133669 2020-03-09T00:25:50.447 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.447 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.447 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.447 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.808711+0000", 2020-03-09T00:25:50.447 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0010978260000000001 2020-03-09T00:25:50.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.448 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.448 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.809808+0000", 2020-03-09T00:25:50.448 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074316110000000003 2020-03-09T00:25:50.448 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.448 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817240+0000", 2020-03-09T00:25:50.449 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010739297 2020-03-09T00:25:50.449 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.449 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.449 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.449 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.827979+0000", 2020-03-09T00:25:50.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7324999999999996e-05 2020-03-09T00:25:50.450 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.450 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.450 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.450 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828057+0000", 2020-03-09T00:25:50.450 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99725236100000003 2020-03-09T00:25:50.450 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.451 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.451 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.451 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.451 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.451 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:399 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~344064] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:50.451 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:50.451 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.536539931999997, 2020-03-09T00:25:50.452 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99540426400000004, 2020-03-09T00:25:50.452 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.452 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.452 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.452 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.452 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.452 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 399 2020-03-09T00:25:50.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.453 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.453 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.453 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.453 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:50.453 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.453 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.454 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.832905+0000", 2020-03-09T00:25:50.454 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00031358200000000002 2020-03-09T00:25:50.454 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.454 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.454 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833219+0000", 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.4809999999999995e-06 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833229+0000", 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.9320000000000008e-06 2020-03-09T00:25:50.455 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.456 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.456 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833238+0000", 2020-03-09T00:25:50.456 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.0749999999999999e-06 2020-03-09T00:25:50.456 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.456 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.456 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833243+0000", 2020-03-09T00:25:50.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1554000000000001e-05 2020-03-09T00:25:50.457 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.457 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.457 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.457 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833294+0000", 2020-03-09T00:25:50.457 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4169999999999999e-06 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833297+0000", 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.979004659 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.458 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.459 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812301+0000", 2020-03-09T00:25:50.459 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013289199999999999 2020-03-09T00:25:50.459 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.459 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.459 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.459 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.812434+0000", 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00098429000000000008 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.813418+0000", 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0052416019999999997 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.460 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.818660+0000", 2020-03-09T00:25:50.461 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0095537799999999996 2020-03-09T00:25:50.461 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.461 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.461 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.461 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828214+0000", 2020-03-09T00:25:50.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6000000000000002e-05 2020-03-09T00:25:50.462 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.462 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.462 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.462 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828310+0000", 2020-03-09T00:25:50.462 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99540426400000004 2020-03-09T00:25:50.462 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.463 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.463 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.463 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.463 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.463 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:400 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:50.463 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:50.463 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.536166330999997, 2020-03-09T00:25:50.464 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99532314300000002, 2020-03-09T00:25:50.464 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.464 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.464 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.464 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.465 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.465 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 400 2020-03-09T00:25:50.465 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.465 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.466 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.466 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:50.466 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.467 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833279+0000", 2020-03-09T00:25:50.467 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9441999999999999e-05 2020-03-09T00:25:50.467 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.467 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.467 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833319+0000", 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.421e-06 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833325+0000", 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0309999999999998e-06 2020-03-09T00:25:50.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.469 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833333+0000", 2020-03-09T00:25:50.469 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6189999999999998e-06 2020-03-09T00:25:50.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.469 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833336+0000", 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.5015999999999998e-05 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833371+0000", 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.305e-06 2020-03-09T00:25:50.470 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.471 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:50.471 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833372+0000", 2020-03-09T00:25:50.471 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.98258547500000004 2020-03-09T00:25:50.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.471 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.472 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.815957+0000", 2020-03-09T00:25:50.472 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000143335 2020-03-09T00:25:50.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.472 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.472 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816101+0000", 2020-03-09T00:25:50.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00076297400000000003 2020-03-09T00:25:50.473 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.473 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.473 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.473 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.816864+0000", 2020-03-09T00:25:50.473 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001798628 2020-03-09T00:25:50.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.474 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.474 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.818662+0000", 2020-03-09T00:25:50.474 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0098116109999999996 2020-03-09T00:25:50.474 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.474 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.475 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.475 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828474+0000", 2020-03-09T00:25:50.475 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000128286 2020-03-09T00:25:50.475 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.475 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.475 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.475 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.828602+0000", 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99532314300000002 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:401 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:50.476 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 46.536085303999997, 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99818093799999996, 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.477 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 401 2020-03-09T00:25:50.478 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.478 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.478 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.478 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.478 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833360+0000", 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3272e-05 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.479 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833383+0000", 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2030000000000001e-06 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833387+0000", 2020-03-09T00:25:50.480 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0190000000000006e-06 2020-03-09T00:25:50.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.481 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.481 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833395+0000", 2020-03-09T00:25:50.481 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3800000000000001e-06 2020-03-09T00:25:50.481 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.481 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833397+0000", 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9891000000000001e-05 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833437+0000", 2020-03-09T00:25:50.482 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.564e-06 2020-03-09T00:25:50.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.483 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for peered", 2020-03-09T00:25:50.483 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.833438+0000", 2020-03-09T00:25:50.483 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.98366688700000005 2020-03-09T00:25:50.483 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.483 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817105+0000", 2020-03-09T00:25:50.484 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000121193 2020-03-09T00:25:50.484 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.484 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.484 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:50.484 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.817227+0000", 2020-03-09T00:25:50.485 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011959001 2020-03-09T00:25:50.485 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.485 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.485 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.485 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.829186+0000", 2020-03-09T00:25:50.486 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012604700000000001 2020-03-09T00:25:50.486 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.487 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.487 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.829312+0000", 2020-03-09T00:25:50.487 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0022294810000000002 2020-03-09T00:25:50.487 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.487 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.488 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.488 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.831541+0000", 2020-03-09T00:25:50.488 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.99818093799999996 2020-03-09T00:25:50.488 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.488 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.488 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.489 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.489 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.489 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs2 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:50.489 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:50.489 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 35.565986389999999, 2020-03-09T00:25:50.489 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019419130999999999, 2020-03-09T00:25:50.489 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.490 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.803459+0000", 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0024772549999999998 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805936+0000", 2020-03-09T00:25:50.491 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6125000000000001e-05 2020-03-09T00:25:50.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.492 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.492 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805962+0000", 2020-03-09T00:25:50.492 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5769000000000001e-05 2020-03-09T00:25:50.492 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.492 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805978+0000", 2020-03-09T00:25:50.493 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.105e-06 2020-03-09T00:25:50.493 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.493 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.493 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.493 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805984+0000", 2020-03-09T00:25:50.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7604e-05 2020-03-09T00:25:50.494 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.494 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.494 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.494 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.806032+0000", 2020-03-09T00:25:50.494 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2700999999999999e-05 2020-03-09T00:25:50.494 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.806045+0000", 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016799584999999999 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.495 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.822844+0000", 2020-03-09T00:25:50.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3986999999999998e-05 2020-03-09T00:25:50.496 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.496 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.496 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.496 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.822878+0000", 2020-03-09T00:25:50.496 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.019419130999999999 2020-03-09T00:25:50.496 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.497 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.497 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.497 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.497 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.497 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:564 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~507904] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:50.497 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:50.497 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.429933276, 2020-03-09T00:25:50.498 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016199960999999999, 2020-03-09T00:25:50.498 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.498 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.498 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.498 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.498 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.498 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 564 2020-03-09T00:25:50.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.499 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.499 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.499 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.499 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:50.499 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.499 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.939512+0000", 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00049015199999999999 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.500 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940002+0000", 2020-03-09T00:25:50.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0630000000000006e-06 2020-03-09T00:25:50.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.501 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.501 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.501 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940010+0000", 2020-03-09T00:25:50.501 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5739999999999998e-05 2020-03-09T00:25:50.501 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.502 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.502 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940026+0000", 2020-03-09T00:25:50.502 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.8399999999999997e-06 2020-03-09T00:25:50.502 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.502 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.502 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.503 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.940033+0000", 2020-03-09T00:25:50.503 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035392219999999999 2020-03-09T00:25:50.503 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.503 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.503 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.503 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943572+0000", 2020-03-09T00:25:50.503 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.8063e-05 2020-03-09T00:25:50.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.504 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.504 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.504 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943660+0000", 2020-03-09T00:25:50.504 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00097790300000000002 2020-03-09T00:25:50.504 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.504 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944638+0000", 2020-03-09T00:25:50.505 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039951850000000001 2020-03-09T00:25:50.505 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.505 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.505 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.505 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948633+0000", 2020-03-09T00:25:50.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0067926059999999996 2020-03-09T00:25:50.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.506 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.506 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.506 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955426+0000", 2020-03-09T00:25:50.506 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00028618699999999998 2020-03-09T00:25:50.506 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.507 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.507 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.507 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955712+0000", 2020-03-09T00:25:50.507 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016199960999999999 2020-03-09T00:25:50.507 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.507 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.507 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.508 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.508 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.508 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:565 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~622592] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:50.508 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:50.508 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.425790331000002, 2020-03-09T00:25:50.508 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016407618999999998, 2020-03-09T00:25:50.508 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.509 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.509 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.509 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.509 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.509 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 565 2020-03-09T00:25:50.509 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.510 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.510 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.510 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.510 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:50.510 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.510 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.510 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.943655+0000", 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00060946899999999996 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944265+0000", 2020-03-09T00:25:50.511 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.004e-05 2020-03-09T00:25:50.512 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.512 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.512 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.512 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944275+0000", 2020-03-09T00:25:50.512 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7229000000000001e-05 2020-03-09T00:25:50.513 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.513 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.513 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944292+0000", 2020-03-09T00:25:50.513 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4939999999999997e-06 2020-03-09T00:25:50.513 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.513 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944299+0000", 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0046037999999999999 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.948903+0000", 2020-03-09T00:25:50.514 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000139479 2020-03-09T00:25:50.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.515 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.515 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.515 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.949043+0000", 2020-03-09T00:25:50.515 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0011480489999999999 2020-03-09T00:25:50.515 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.515 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.516 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.516 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950191+0000", 2020-03-09T00:25:50.516 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055348039999999999 2020-03-09T00:25:50.516 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.516 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.516 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.516 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955725+0000", 2020-03-09T00:25:50.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041247109999999997 2020-03-09T00:25:50.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.517 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.517 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.517 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.959850+0000", 2020-03-09T00:25:50.517 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000212544 2020-03-09T00:25:50.517 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.518 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.518 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.518 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960063+0000", 2020-03-09T00:25:50.518 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016407618999999998 2020-03-09T00:25:50.518 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.518 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.518 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.519 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.519 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.519 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:566 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [append 0~32768] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:50.519 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:50.519 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.425081016999997, 2020-03-09T00:25:50.519 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016519618999999999, 2020-03-09T00:25:50.520 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.520 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.520 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.520 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.520 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.520 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 566 2020-03-09T00:25:50.520 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.521 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.521 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.521 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:50.521 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.521 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.521 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944364+0000", 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5429e-05 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944430+0000", 2020-03-09T00:25:50.522 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2100000000000003e-06 2020-03-09T00:25:50.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.523 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.523 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944434+0000", 2020-03-09T00:25:50.523 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.4159999999999998e-06 2020-03-09T00:25:50.523 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.523 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944441+0000", 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.3349999999999998e-06 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944445+0000", 2020-03-09T00:25:50.524 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0093476589999999995 2020-03-09T00:25:50.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.525 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.525 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953792+0000", 2020-03-09T00:25:50.525 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014473700000000001 2020-03-09T00:25:50.525 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.525 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953937+0000", 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00083984299999999995 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.954777+0000", 2020-03-09T00:25:50.526 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.002244401 2020-03-09T00:25:50.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.527 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.527 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957021+0000", 2020-03-09T00:25:50.527 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0035682729999999998 2020-03-09T00:25:50.527 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.527 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960590+0000", 2020-03-09T00:25:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00029431599999999998 2020-03-09T00:25:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.528 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960884+0000", 2020-03-09T00:25:50.529 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016519618999999999 2020-03-09T00:25:50.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.529 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.529 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.529 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.529 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:567 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [setxattr _header (57) in=7b] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:50.530 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:50.530 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.424971311, 2020-03-09T00:25:50.530 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017056153000000001, 2020-03-09T00:25:50.530 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.530 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.530 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.530 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 567 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:50.531 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.532 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.532 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944474+0000", 2020-03-09T00:25:50.532 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8381999999999999e-05 2020-03-09T00:25:50.532 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.532 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.533 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.533 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944513+0000", 2020-03-09T00:25:50.533 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.1709999999999998e-06 2020-03-09T00:25:50.533 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.533 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.533 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.533 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944516+0000", 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7849999999999992e-06 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944523+0000", 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2550000000000002e-06 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.534 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944527+0000", 2020-03-09T00:25:50.535 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011392539 2020-03-09T00:25:50.535 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.535 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.535 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.535 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.955919+0000", 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013716499999999999 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956056+0000", 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00057513499999999995 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.536 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956632+0000", 2020-03-09T00:25:50.537 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00045027700000000001 2020-03-09T00:25:50.537 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.537 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.537 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.537 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957082+0000", 2020-03-09T00:25:50.538 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0042876219999999996 2020-03-09T00:25:50.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.538 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.538 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.538 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.961369+0000", 2020-03-09T00:25:50.538 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000160822 2020-03-09T00:25:50.538 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.539 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.539 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.539 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.961530+0000", 2020-03-09T00:25:50.539 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.017056153000000001 2020-03-09T00:25:50.539 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.539 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.539 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.540 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.540 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.540 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:568 3.7s0 3:e0ed5a0f:::smithi10428818-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:50.540 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:50.540 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 32.424877840000001, 2020-03-09T00:25:50.540 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020843882000000001, 2020-03-09T00:25:50.540 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.541 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:50.541 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.541 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.541 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.541 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 568 2020-03-09T00:25:50.541 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.541 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.542 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944568+0000", 2020-03-09T00:25:50.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8635999999999999e-05 2020-03-09T00:25:50.543 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.543 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.543 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944626+0000", 2020-03-09T00:25:50.543 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2609999999999999e-06 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944632+0000", 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4993e-05 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.544 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944647+0000", 2020-03-09T00:25:50.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9439999999999996e-06 2020-03-09T00:25:50.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.545 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.545 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.545 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.944652+0000", 2020-03-09T00:25:50.545 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013280882000000001 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.957933+0000", 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012477000000000001 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.546 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:50.547 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.958058+0000", 2020-03-09T00:25:50.547 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0044866419999999999 2020-03-09T00:25:50.547 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.547 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.547 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.547 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962545+0000", 2020-03-09T00:25:50.547 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000120438 2020-03-09T00:25:50.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.548 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.548 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.962665+0000", 2020-03-09T00:25:50.548 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027463159999999999 2020-03-09T00:25:50.548 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.548 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.549 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.549 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.965411+0000", 2020-03-09T00:25:50.549 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.020843882000000001 2020-03-09T00:25:50.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.549 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.549 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.549 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.550 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.550 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:672 3.6s0 3:6178963a:::smithi10428818-25:head [copy-from ver 45] snapc 2c=[2c,2b,2a,24,23] ondisk+write+known_if_redirected e80)", 2020-03-09T00:25:50.550 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:29.806423+0000", 2020-03-09T00:25:50.550 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 20.563022911000001, 2020-03-09T00:25:50.550 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.029706512000000001, 2020-03-09T00:25:50.550 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.550 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 672 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.551 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806423+0000", 2020-03-09T00:25:50.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.552 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.552 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.552 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.552 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806423+0000", 2020-03-09T00:25:50.552 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.2702999999999998e-05 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806495+0000", 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0570999999999999e-05 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.553 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806506+0000", 2020-03-09T00:25:50.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.588e-05 2020-03-09T00:25:50.554 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.554 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.554 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.554 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806522+0000", 2020-03-09T00:25:50.554 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.0649999999999999e-06 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806530+0000", 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8251000000000003e-05 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.555 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806578+0000", 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00014559299999999999 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806724+0000", 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.013694415 2020-03-09T00:25:50.556 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.557 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.557 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.820418+0000", 2020-03-09T00:25:50.557 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0089145890000000005 2020-03-09T00:25:50.557 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.557 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.829333+0000", 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0066116930000000001 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.835944+0000", 2020-03-09T00:25:50.558 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000184752 2020-03-09T00:25:50.559 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.559 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.559 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.559 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.836129+0000", 2020-03-09T00:25:50.559 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.029706512000000001 2020-03-09T00:25:50.559 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.559 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.560 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.560 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.560 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.560 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:673 3.6s0 3:6178963a:::smithi10428818-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e80)", 2020-03-09T00:25:50.560 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:29.806577+0000", 2020-03-09T00:25:50.560 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 20.562868590000001, 2020-03-09T00:25:50.560 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.030424049000000002, 2020-03-09T00:25:50.561 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.561 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:50.561 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.561 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.561 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.561 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 673 2020-03-09T00:25:50.561 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.562 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.562 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.562 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806577+0000", 2020-03-09T00:25:50.562 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.562 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.562 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806577+0000", 2020-03-09T00:25:50.563 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.4972999999999998e-05 2020-03-09T00:25:50.563 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.563 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.563 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.563 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806622+0000", 2020-03-09T00:25:50.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5189999999999998e-06 2020-03-09T00:25:50.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.564 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.564 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.564 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806628+0000", 2020-03-09T00:25:50.564 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2245e-05 2020-03-09T00:25:50.564 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.565 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.565 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.565 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806641+0000", 2020-03-09T00:25:50.565 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.78e-06 2020-03-09T00:25:50.565 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.566 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.566 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.566 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806645+0000", 2020-03-09T00:25:50.566 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00025891500000000002 2020-03-09T00:25:50.566 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.567 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.567 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.567 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806904+0000", 2020-03-09T00:25:50.567 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4018999999999999e-05 2020-03-09T00:25:50.567 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.806958+0000", 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022279448 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.568 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.829238+0000", 2020-03-09T00:25:50.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7843999999999994e-05 2020-03-09T00:25:50.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.569 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.569 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:50.569 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.829326+0000", 2020-03-09T00:25:50.569 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074501960000000001 2020-03-09T00:25:50.569 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.836776+0000", 2020-03-09T00:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3979999999999994e-05 2020-03-09T00:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.570 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.571 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.836840+0000", 2020-03-09T00:25:50.571 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00016113 2020-03-09T00:25:50.571 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.571 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.571 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.571 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:29.837001+0000", 2020-03-09T00:25:50.571 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.030424049000000002 2020-03-09T00:25:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:762 3.6s0 3:6d582d75:::smithi10428818-10:head [append 0~622592] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:50.572 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:34.890681+0000", 2020-03-09T00:25:50.573 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.478764212, 2020-03-09T00:25:50.573 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016558272999999998, 2020-03-09T00:25:50.574 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.574 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:50.574 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.574 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.574 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.574 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 762 2020-03-09T00:25:50.574 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.575 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.575 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.575 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.575 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.890681+0000", 2020-03-09T00:25:50.575 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.575 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.575 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.890681+0000", 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00064092800000000003 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891322+0000", 2020-03-09T00:25:50.576 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1939999999999998e-06 2020-03-09T00:25:50.577 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.577 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.577 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.577 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891330+0000", 2020-03-09T00:25:50.577 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5204999999999999e-05 2020-03-09T00:25:50.577 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.577 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.578 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.578 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891346+0000", 2020-03-09T00:25:50.578 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9970000000000001e-06 2020-03-09T00:25:50.578 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.578 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.578 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.579 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.891353+0000", 2020-03-09T00:25:50.579 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037718790000000001 2020-03-09T00:25:50.579 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.579 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.579 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.579 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.895124+0000", 2020-03-09T00:25:50.579 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00013079899999999999 2020-03-09T00:25:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.895255+0000", 2020-03-09T00:25:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001217615 2020-03-09T00:25:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.581 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:50.581 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.896473+0000", 2020-03-09T00:25:50.581 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0071182620000000002 2020-03-09T00:25:50.581 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.581 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.581 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:50.581 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.903591+0000", 2020-03-09T00:25:50.582 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.003383371 2020-03-09T00:25:50.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.582 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.582 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "commit_sent", 2020-03-09T00:25:50.582 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.906974+0000", 2020-03-09T00:25:50.582 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00026502299999999997 2020-03-09T00:25:50.582 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.907239+0000", 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016558272999999998 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.583 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.584 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.584 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "osd_op(client.4751.0:765 3.6s0 3:6d582d75:::smithi10428818-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e85)", 2020-03-09T00:25:50.584 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:34.893621+0000", 2020-03-09T00:25:50.584 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 15.475824362999999, 2020-03-09T00:25:50.584 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016719357000000001, 2020-03-09T00:25:50.584 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:50.584 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "started", 2020-03-09T00:25:50.585 INFO:teuthology.orchestra.run.smithi139.stdout: "client_info": { 2020-03-09T00:25:50.585 INFO:teuthology.orchestra.run.smithi139.stdout: "client": "client.4751", 2020-03-09T00:25:50.585 INFO:teuthology.orchestra.run.smithi139.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:50.586 INFO:teuthology.orchestra.run.smithi139.stdout: "tid": 765 2020-03-09T00:25:50.586 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.586 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:50.587 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.587 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:50.587 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893621+0000", 2020-03-09T00:25:50.587 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:50.587 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.587 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.587 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:50.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893621+0000", 2020-03-09T00:25:50.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3843e-05 2020-03-09T00:25:50.588 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.588 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.588 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:50.588 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893665+0000", 2020-03-09T00:25:50.588 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8669999999999999e-06 2020-03-09T00:25:50.589 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.589 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.589 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:50.589 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893671+0000", 2020-03-09T00:25:50.589 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5889999999999999e-05 2020-03-09T00:25:50.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.590 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.590 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:50.590 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893687+0000", 2020-03-09T00:25:50.590 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.2579999999999997e-06 2020-03-09T00:25:50.590 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.590 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.893691+0000", 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010884733000000001 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.904576+0000", 2020-03-09T00:25:50.591 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000100896 2020-03-09T00:25:50.592 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.592 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.592 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:50.592 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.904677+0000", 2020-03-09T00:25:50.592 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0041684770000000003 2020-03-09T00:25:50.592 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.592 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.593 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:50.593 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.908845+0000", 2020-03-09T00:25:50.593 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00012772599999999999 2020-03-09T00:25:50.593 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.593 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.593 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "started", 2020-03-09T00:25:50.593 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.908973+0000", 2020-03-09T00:25:50.594 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0013676669999999999 2020-03-09T00:25:50.594 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:50.594 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:50.594 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:50.594 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:34.910340+0000", 2020-03-09T00:25:50.594 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.016719357000000001 2020-03-09T00:25:50.594 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.595 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.595 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.595 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:50.595 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:50.595 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:50.597 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-03-09T00:25:50.597 INFO:tasks.ceph.osd.11.smithi104.stderr:2020-03-09T00:25:50.595+0000 7fb6fc7d6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 28008) UID: 0 2020-03-09T00:25:50.698 INFO:tasks.ceph.osd.1.smithi050.stderr:2020-03-09T00:25:50.696+0000 7fa1e7d13700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27622) UID: 0 2020-03-09T00:25:50.734 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:50.734 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:50.734 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:50.734 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:50.745 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-03-09T00:25:50.798 INFO:tasks.ceph.osd.4.smithi148.stderr:2020-03-09T00:25:50.797+0000 7fd6c839e700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27358) UID: 0 2020-03-09T00:25:50.896 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'unique_pool_0' removed 2020-03-09T00:25:50.899 INFO:tasks.ceph.osd.12.smithi148.stderr:2020-03-09T00:25:50.897+0000 7f442d72d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 27361) UID: 0 2020-03-09T00:25:50.903 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd set noscrub 2020-03-09T00:25:50.910 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2020-03-09T00:25:50.921 INFO:tasks.thrashosds:joining thrashosds 2020-03-09T00:25:50.921 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2020-03-09T00:25:50.921 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:50.922 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:50.922 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:50.922 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:50.922 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:50.924 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-03-09T00:25:50.999 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2020-03-09T00:25:51.050 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.197227542999997, 2020-03-09T00:25:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057419319999999999, 2020-03-09T00:25:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850400+0000", 2020-03-09T00:25:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056030900000000005 2020-03-09T00:25:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850960+0000", 2020-03-09T00:25:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9046e-05 2020-03-09T00:25:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850979+0000", 2020-03-09T00:25:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0473e-05 2020-03-09T00:25:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850990+0000", 2020-03-09T00:25:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.198e-06 2020-03-09T00:25:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.850995+0000", 2020-03-09T00:25:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123569 2020-03-09T00:25:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.851119+0000", 2020-03-09T00:25:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4375e-05 2020-03-09T00:25:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.851133+0000", 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049526300000000004 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856086+0000", 2020-03-09T00:25:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6332000000000002e-05 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856142+0000", 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057419319999999999 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.183639583000001, 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060463440000000004, 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863988+0000", 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088386200000000002 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864872+0000", 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2838e-05 2020-03-09T00:25:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864905+0000", 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9593e-05 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864924+0000", 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8414999999999999e-05 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.864943+0000", 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000227028 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865170+0000", 2020-03-09T00:25:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7546000000000001e-05 2020-03-09T00:25:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.865187+0000", 2020-03-09T00:25:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047657489999999997 2020-03-09T00:25:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.869953+0000", 2020-03-09T00:25:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1312999999999999e-05 2020-03-09T00:25:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.870034+0000", 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060463440000000004 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/32 ECSubWrite(tid=22, reqid=client.4751.0:275, at_version=48'12, trim_to=0'0, roll_forward_to=38'10))", 2020-03-09T00:25:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.259701862999997, 2020-03-09T00:25:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080719139999999995, 2020-03-09T00:25:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787926+0000", 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013854899999999999 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788064+0000", 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9998000000000001e-05 2020-03-09T00:25:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788084+0000", 2020-03-09T00:25:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2672e-05 2020-03-09T00:25:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788097+0000", 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5619999999999998e-06 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788104+0000", 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0817999999999999e-05 2020-03-09T00:25:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788165+0000", 2020-03-09T00:25:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9372999999999999e-05 2020-03-09T00:25:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788185+0000", 2020-03-09T00:25:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077469949999999996 2020-03-09T00:25:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795932+0000", 2020-03-09T00:25:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5946999999999997e-05 2020-03-09T00:25:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795998+0000", 2020-03-09T00:25:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080719139999999995 2020-03-09T00:25:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.230540781000002, 2020-03-09T00:25:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052768320000000004, 2020-03-09T00:25:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:51.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817087+0000", 2020-03-09T00:25:51.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040852899999999998 2020-03-09T00:25:51.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817495+0000", 2020-03-09T00:25:51.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1301000000000002e-05 2020-03-09T00:25:51.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817517+0000", 2020-03-09T00:25:51.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4061e-05 2020-03-09T00:25:51.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817531+0000", 2020-03-09T00:25:51.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0720000000000001e-06 2020-03-09T00:25:51.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817535+0000", 2020-03-09T00:25:51.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012169500000000001 2020-03-09T00:25:51.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817656+0000", 2020-03-09T00:25:51.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3953e-05 2020-03-09T00:25:51.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817670+0000", 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046400950000000003 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822310+0000", 2020-03-09T00:25:51.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3125999999999999e-05 2020-03-09T00:25:51.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822364+0000", 2020-03-09T00:25:51.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052768320000000004 2020-03-09T00:25:51.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.084 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.085 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.085 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es2 54/32 ECSubWrite(tid=77, reqid=client.4751.0:415, at_version=54'44, trim_to=0'0, roll_forward_to=54'42))", 2020-03-09T00:25:51.085 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:51.085 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.844685298000002, 2020-03-09T00:25:51.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066307969999999999, 2020-03-09T00:25:51.086 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.086 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.086 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:51.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.202942+0000", 2020-03-09T00:25:51.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055621700000000002 2020-03-09T00:25:51.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203498+0000", 2020-03-09T00:25:51.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0248999999999999e-05 2020-03-09T00:25:51.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203519+0000", 2020-03-09T00:25:51.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3159000000000001e-05 2020-03-09T00:25:51.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203532+0000", 2020-03-09T00:25:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8510000000000004e-06 2020-03-09T00:25:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203538+0000", 2020-03-09T00:25:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1504e-05 2020-03-09T00:25:51.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203589+0000", 2020-03-09T00:25:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5809000000000001e-05 2020-03-09T00:25:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.203605+0000", 2020-03-09T00:25:51.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059129430000000004 2020-03-09T00:25:51.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.209518+0000", 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5065000000000002e-05 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:04.209573+0000", 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066307969999999999 2020-03-09T00:25:51.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.094 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.094 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:51.094 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:51.094 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.133519468000003, 2020-03-09T00:25:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094365569999999999, 2020-03-09T00:25:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:51.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.914108+0000", 2020-03-09T00:25:51.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011685599999999999 2020-03-09T00:25:51.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915277+0000", 2020-03-09T00:25:51.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3901999999999999e-05 2020-03-09T00:25:51.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915301+0000", 2020-03-09T00:25:51.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4554e-05 2020-03-09T00:25:51.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915315+0000", 2020-03-09T00:25:51.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8169999999999999e-06 2020-03-09T00:25:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915323+0000", 2020-03-09T00:25:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2163999999999998e-05 2020-03-09T00:25:51.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915365+0000", 2020-03-09T00:25:51.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4710000000000001e-06 2020-03-09T00:25:51.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.915373+0000", 2020-03-09T00:25:51.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081172520000000001 2020-03-09T00:25:51.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923490+0000", 2020-03-09T00:25:51.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4836999999999997e-05 2020-03-09T00:25:51.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.923545+0000", 2020-03-09T00:25:51.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0094365569999999999 2020-03-09T00:25:51.102 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.103 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.103 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.103 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs2 68/67 ECSubWrite(tid=91, reqid=client.4751.0:580, at_version=68'59, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.103 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.967188+0000", 2020-03-09T00:25:51.103 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.080439534999996, 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058375959999999996, 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967188+0000", 2020-03-09T00:25:51.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967188+0000", 2020-03-09T00:25:51.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00078212699999999997 2020-03-09T00:25:51.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967970+0000", 2020-03-09T00:25:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4190999999999999e-05 2020-03-09T00:25:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.967994+0000", 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6008e-05 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968010+0000", 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6010000000000002e-06 2020-03-09T00:25:51.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968016+0000", 2020-03-09T00:25:51.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3309999999999997e-05 2020-03-09T00:25:51.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968059+0000", 2020-03-09T00:25:51.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3348e-05 2020-03-09T00:25:51.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.968073+0000", 2020-03-09T00:25:51.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048942650000000001 2020-03-09T00:25:51.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.972967+0000", 2020-03-09T00:25:51.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8746000000000003e-05 2020-03-09T00:25:51.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973026+0000", 2020-03-09T00:25:51.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058375959999999996 2020-03-09T00:25:51.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.112 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.112 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s1 68/67 ECSubWrite(tid=237, reqid=client.4751.0:583, at_version=68'129, trim_to=0'0, roll_forward_to=68'126))", 2020-03-09T00:25:51.112 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.973084+0000", 2020-03-09T00:25:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.07454396, 2020-03-09T00:25:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077488779999999998, 2020-03-09T00:25:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973084+0000", 2020-03-09T00:25:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973084+0000", 2020-03-09T00:25:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020823799999999999 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973292+0000", 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9747000000000002e-05 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973322+0000", 2020-03-09T00:25:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1814000000000002e-05 2020-03-09T00:25:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973343+0000", 2020-03-09T00:25:51.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8770000000000005e-06 2020-03-09T00:25:51.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973353+0000", 2020-03-09T00:25:51.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7701000000000002e-05 2020-03-09T00:25:51.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973411+0000", 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5305999999999999e-05 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.973426+0000", 2020-03-09T00:25:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073632940000000003 2020-03-09T00:25:51.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980790+0000", 2020-03-09T00:25:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2901000000000001e-05 2020-03-09T00:25:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980832+0000", 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077488779999999998 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.2s5 68/48 ECSubWrite(tid=36, reqid=client.4751.0:587, at_version=68'43, trim_to=0'0, roll_forward_to=67'41))", 2020-03-09T00:25:51.121 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.979226+0000", 2020-03-09T00:25:51.122 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.068401129000002, 2020-03-09T00:25:51.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067534049999999997, 2020-03-09T00:25:51.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.122 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979226+0000", 2020-03-09T00:25:51.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979226+0000", 2020-03-09T00:25:51.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013291099999999999 2020-03-09T00:25:51.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979359+0000", 2020-03-09T00:25:51.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4303000000000001e-05 2020-03-09T00:25:51.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979384+0000", 2020-03-09T00:25:51.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8671000000000001e-05 2020-03-09T00:25:51.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979402+0000", 2020-03-09T00:25:51.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6800000000000005e-06 2020-03-09T00:25:51.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979412+0000", 2020-03-09T00:25:51.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6808999999999998e-05 2020-03-09T00:25:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979469+0000", 2020-03-09T00:25:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2595000000000001e-05 2020-03-09T00:25:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979481+0000", 2020-03-09T00:25:51.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064418380000000001 2020-03-09T00:25:51.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985923+0000", 2020-03-09T00:25:51.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6598000000000003e-05 2020-03-09T00:25:51.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985980+0000", 2020-03-09T00:25:51.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067534049999999997 2020-03-09T00:25:51.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.130 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.130 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.130 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 68/67 ECSubWrite(tid=101, reqid=client.4751.0:592, at_version=68'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.130 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.989023+0000", 2020-03-09T00:25:51.130 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.058604928999998, 2020-03-09T00:25:51.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079003479999999997, 2020-03-09T00:25:51.131 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.131 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.131 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989023+0000", 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989023+0000", 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064198599999999999 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989665+0000", 2020-03-09T00:25:51.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6575e-05 2020-03-09T00:25:51.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989681+0000", 2020-03-09T00:25:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0254e-05 2020-03-09T00:25:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989701+0000", 2020-03-09T00:25:51.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8129999999999998e-06 2020-03-09T00:25:51.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989708+0000", 2020-03-09T00:25:51.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7768000000000001e-05 2020-03-09T00:25:51.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989756+0000", 2020-03-09T00:25:51.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5208999999999999e-05 2020-03-09T00:25:51.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.989781+0000", 2020-03-09T00:25:51.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070914699999999999 2020-03-09T00:25:51.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996873+0000", 2020-03-09T00:25:51.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0272999999999998e-05 2020-03-09T00:25:51.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996923+0000", 2020-03-09T00:25:51.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079003479999999997 2020-03-09T00:25:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 68/67 ECSubWrite(tid=102, reqid=client.4751.0:593, at_version=68'24, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.139 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.993168+0000", 2020-03-09T00:25:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.054459700000002, 2020-03-09T00:25:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068995819999999996, 2020-03-09T00:25:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993168+0000", 2020-03-09T00:25:51.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993168+0000", 2020-03-09T00:25:51.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154976 2020-03-09T00:25:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993323+0000", 2020-03-09T00:25:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6016e-05 2020-03-09T00:25:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993349+0000", 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8249000000000001e-05 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993367+0000", 2020-03-09T00:25:51.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2140000000000001e-06 2020-03-09T00:25:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993373+0000", 2020-03-09T00:25:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016447 2020-03-09T00:25:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993538+0000", 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5218000000000001e-05 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.993553+0000", 2020-03-09T00:25:51.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064617299999999997 2020-03-09T00:25:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:18.000015+0000", 2020-03-09T00:25:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2709e-05 2020-03-09T00:25:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:18.000067+0000", 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068995819999999996 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.148 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s3 75/67 ECSubWrite(tid=93, reqid=client.4751.0:637, at_version=75'8, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.148 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:24.810094+0000", 2020-03-09T00:25:51.148 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.237534046, 2020-03-09T00:25:51.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015640494000000001, 2020-03-09T00:25:51.148 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.148 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.148 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.810094+0000", 2020-03-09T00:25:51.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.810094+0000", 2020-03-09T00:25:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022108739999999998 2020-03-09T00:25:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812304+0000", 2020-03-09T00:25:51.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4171999999999997e-05 2020-03-09T00:25:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812339+0000", 2020-03-09T00:25:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2938e-05 2020-03-09T00:25:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812351+0000", 2020-03-09T00:25:51.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.221e-06 2020-03-09T00:25:51.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812359+0000", 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5735000000000007e-05 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812424+0000", 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5446000000000001e-05 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.812440+0000", 2020-03-09T00:25:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013249169999999999 2020-03-09T00:25:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.825689+0000", 2020-03-09T00:25:51.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4938000000000002e-05 2020-03-09T00:25:51.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:24.825734+0000", 2020-03-09T00:25:51.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015640494000000001 2020-03-09T00:25:51.155 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.156 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.156 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.156 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:51.156 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:29.821237+0000", 2020-03-09T00:25:51.156 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.226390465000001, 2020-03-09T00:25:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089333529999999998, 2020-03-09T00:25:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821237+0000", 2020-03-09T00:25:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821237+0000", 2020-03-09T00:25:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010328519999999999 2020-03-09T00:25:51.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822270+0000", 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2592999999999999e-05 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822293+0000", 2020-03-09T00:25:51.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2020-03-09T00:25:51.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822304+0000", 2020-03-09T00:25:51.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4410000000000001e-06 2020-03-09T00:25:51.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822309+0000", 2020-03-09T00:25:51.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2065999999999999e-05 2020-03-09T00:25:51.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822361+0000", 2020-03-09T00:25:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7379000000000001e-05 2020-03-09T00:25:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822378+0000", 2020-03-09T00:25:51.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077357290000000002 2020-03-09T00:25:51.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.830114+0000", 2020-03-09T00:25:51.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6314999999999999e-05 2020-03-09T00:25:51.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.830170+0000", 2020-03-09T00:25:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089333529999999998 2020-03-09T00:25:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s3 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.884858+0000", 2020-03-09T00:25:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.162770036000001, 2020-03-09T00:25:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018966332999999998, 2020-03-09T00:25:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.165 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.884858+0000", 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.884858+0000", 2020-03-09T00:25:51.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039268819999999996 2020-03-09T00:25:51.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888784+0000", 2020-03-09T00:25:51.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9009999999999999e-05 2020-03-09T00:25:51.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888803+0000", 2020-03-09T00:25:51.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2874000000000001e-05 2020-03-09T00:25:51.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888816+0000", 2020-03-09T00:25:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8810000000000004e-06 2020-03-09T00:25:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888821+0000", 2020-03-09T00:25:51.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4030999999999999e-05 2020-03-09T00:25:51.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888865+0000", 2020-03-09T00:25:51.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2290000000000003e-06 2020-03-09T00:25:51.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.888873+0000", 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014886276 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.903760+0000", 2020-03-09T00:25:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4150000000000001e-05 2020-03-09T00:25:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.903824+0000", 2020-03-09T00:25:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018966332999999998 2020-03-09T00:25:51.172 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.173 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.173 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.173 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 85/67 ECSubWrite(tid=124, reqid=client.4751.0:762, at_version=85'38, trim_to=0'0, roll_forward_to=85'37))", 2020-03-09T00:25:51.173 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.896639+0000", 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.150988761000001, 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099964359999999992, 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896639+0000", 2020-03-09T00:25:51.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.896639+0000", 2020-03-09T00:25:51.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052756099999999996 2020-03-09T00:25:51.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897166+0000", 2020-03-09T00:25:51.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7646999999999999e-05 2020-03-09T00:25:51.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897184+0000", 2020-03-09T00:25:51.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0543000000000001e-05 2020-03-09T00:25:51.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897195+0000", 2020-03-09T00:25:51.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0690000000000004e-06 2020-03-09T00:25:51.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.897202+0000", 2020-03-09T00:25:51.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034302289999999999 2020-03-09T00:25:51.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900632+0000", 2020-03-09T00:25:51.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.378e-05 2020-03-09T00:25:51.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900646+0000", 2020-03-09T00:25:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059260010000000002 2020-03-09T00:25:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906572+0000", 2020-03-09T00:25:51.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3606000000000001e-05 2020-03-09T00:25:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906635+0000", 2020-03-09T00:25:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099964359999999992 2020-03-09T00:25:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.181 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s2 85/67 ECSubWrite(tid=126, reqid=client.4751.0:763, at_version=85'39, trim_to=0'0, roll_forward_to=85'37))", 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.901520+0000", 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.146107315999998, 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052168550000000003, 2020-03-09T00:25:51.182 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.183 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.183 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901520+0000", 2020-03-09T00:25:51.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901520+0000", 2020-03-09T00:25:51.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000133857 2020-03-09T00:25:51.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901654+0000", 2020-03-09T00:25:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5188999999999999e-05 2020-03-09T00:25:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901679+0000", 2020-03-09T00:25:51.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7649e-05 2020-03-09T00:25:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901697+0000", 2020-03-09T00:25:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0185e-05 2020-03-09T00:25:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.901707+0000", 2020-03-09T00:25:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028222220000000001 2020-03-09T00:25:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.904529+0000", 2020-03-09T00:25:51.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2162e-05 2020-03-09T00:25:51.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.904542+0000", 2020-03-09T00:25:51.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021478690000000002 2020-03-09T00:25:51.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906689+0000", 2020-03-09T00:25:51.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7722000000000002e-05 2020-03-09T00:25:51.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.906737+0000", 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052168550000000003 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.191 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 94/32 ECSubWrite(tid=42, reqid=client.4751.0:911, at_version=94'40, trim_to=0'0, roll_forward_to=84'36))", 2020-03-09T00:25:51.191 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:44.096943+0000", 2020-03-09T00:25:51.191 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.9506844450000003, 2020-03-09T00:25:51.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068740629999999997, 2020-03-09T00:25:51.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.192 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.192 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.096943+0000", 2020-03-09T00:25:51.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.096943+0000", 2020-03-09T00:25:51.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029346299999999999 2020-03-09T00:25:51.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097237+0000", 2020-03-09T00:25:51.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5407e-05 2020-03-09T00:25:51.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097252+0000", 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7456e-05 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097269+0000", 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0971e-05 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097280+0000", 2020-03-09T00:25:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1446999999999995e-05 2020-03-09T00:25:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097352+0000", 2020-03-09T00:25:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1472e-05 2020-03-09T00:25:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097363+0000", 2020-03-09T00:25:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064163149999999997 2020-03-09T00:25:51.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103780+0000", 2020-03-09T00:25:51.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7531999999999999e-05 2020-03-09T00:25:51.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103817+0000", 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068740629999999997 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.200 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 94/32 ECSubWrite(tid=43, reqid=client.4751.0:912, at_version=94'41, trim_to=0'0, roll_forward_to=94'39))", 2020-03-09T00:25:51.201 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:44.097329+0000", 2020-03-09T00:25:51.201 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.950298697, 2020-03-09T00:25:51.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065171760000000004, 2020-03-09T00:25:51.201 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.201 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.202 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097329+0000", 2020-03-09T00:25:51.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097329+0000", 2020-03-09T00:25:51.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015804400000000001 2020-03-09T00:25:51.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097487+0000", 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3022e-05 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097500+0000", 2020-03-09T00:25:51.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0308e-05 2020-03-09T00:25:51.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097510+0000", 2020-03-09T00:25:51.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8109999999999999e-06 2020-03-09T00:25:51.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097516+0000", 2020-03-09T00:25:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025743570000000002 2020-03-09T00:25:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.100090+0000", 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9979999999999992e-06 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.100099+0000", 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037222219999999999 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103822+0000", 2020-03-09T00:25:51.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4414e-05 2020-03-09T00:25:51.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103846+0000", 2020-03-09T00:25:51.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065171760000000004 2020-03-09T00:25:51.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.209 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.209 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 94/32 ECSubWrite(tid=44, reqid=client.4751.0:913, at_version=94'42, trim_to=0'0, roll_forward_to=94'39))", 2020-03-09T00:25:51.210 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:44.097555+0000", 2020-03-09T00:25:51.210 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.9500721900000002, 2020-03-09T00:25:51.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063091939999999997, 2020-03-09T00:25:51.210 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.210 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.210 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097555+0000", 2020-03-09T00:25:51.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097555+0000", 2020-03-09T00:25:51.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9408999999999999e-05 2020-03-09T00:25:51.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097625+0000", 2020-03-09T00:25:51.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2203e-05 2020-03-09T00:25:51.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097637+0000", 2020-03-09T00:25:51.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0651e-05 2020-03-09T00:25:51.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097648+0000", 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0499999999999997e-06 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097654+0000", 2020-03-09T00:25:51.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051808419999999997 2020-03-09T00:25:51.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.102835+0000", 2020-03-09T00:25:51.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5650000000000002e-06 2020-03-09T00:25:51.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.102843+0000", 2020-03-09T00:25:51.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001004039 2020-03-09T00:25:51.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103847+0000", 2020-03-09T00:25:51.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7435000000000002e-05 2020-03-09T00:25:51.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103865+0000", 2020-03-09T00:25:51.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063091939999999997 2020-03-09T00:25:51.219 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.219 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.219 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.219 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.as2 94/32 ECSubWrite(tid=45, reqid=client.4751.0:914, at_version=94'43, trim_to=0'0, roll_forward_to=94'39))", 2020-03-09T00:25:51.220 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:44.097684+0000", 2020-03-09T00:25:51.220 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.9499440090000002, 2020-03-09T00:25:51.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062570550000000001, 2020-03-09T00:25:51.220 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:51.220 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.220 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:51.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:51.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097684+0000", 2020-03-09T00:25:51.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:51.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:51.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097684+0000", 2020-03-09T00:25:51.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3223999999999999e-05 2020-03-09T00:25:51.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:51.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097717+0000", 2020-03-09T00:25:51.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8559999999999996e-06 2020-03-09T00:25:51.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:51.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097726+0000", 2020-03-09T00:25:51.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3000000000000007e-06 2020-03-09T00:25:51.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097735+0000", 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0620000000000001e-06 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.097738+0000", 2020-03-09T00:25:51.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058968220000000003 2020-03-09T00:25:51.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:51.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103635+0000", 2020-03-09T00:25:51.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3070000000000003e-06 2020-03-09T00:25:51.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103643+0000", 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027083800000000002 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103914+0000", 2020-03-09T00:25:51.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6645999999999999e-05 2020-03-09T00:25:51.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:51.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:51.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:51.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:44.103941+0000", 2020-03-09T00:25:51.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062570550000000001 2020-03-09T00:25:51.227 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.228 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.228 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.228 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:51.228 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:51.228 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:51.229 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-03-09T00:25:51.361 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:51.362 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:51.362 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:51.362 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:51.371 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-03-09T00:25:51.374 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2020-03-09T00:25:51.485 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:51.485 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:51.485 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:51.486 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:51.486 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:51.494 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-03-09T00:25:51.619 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:51.619 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:51.619 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.es5 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 57.767048179, 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999, 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.620 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.621 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.621 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.621 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:51.621 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849961+0000", 2020-03-09T00:25:51.622 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00017622699999999999 2020-03-09T00:25:51.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.622 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.622 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850137+0000", 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1974000000000003e-05 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850169+0000", 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.4702000000000002e-05 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850193+0000", 2020-03-09T00:25:51.624 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5790000000000004e-06 2020-03-09T00:25:51.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.624 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.624 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850202+0000", 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013641599999999999 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850338+0000", 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4691999999999999e-05 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.625 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850353+0000", 2020-03-09T00:25:51.626 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011849303 2020-03-09T00:25:51.626 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.626 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.626 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.626 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862202+0000", 2020-03-09T00:25:51.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1013000000000001e-05 2020-03-09T00:25:51.627 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.627 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.627 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.627 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862253+0000", 2020-03-09T00:25:51.627 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012292905999999999 2020-03-09T00:25:51.627 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(osd.10.23:4 3.3s0 3:c607ec57:::smithi10428818-30:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e48)", 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 53.827879553999999, 2020-03-09T00:25:51.628 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999, 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.629 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789129+0000", 2020-03-09T00:25:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.8078999999999999e-05 2020-03-09T00:25:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.630 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789167+0000", 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1499999999999998e-06 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789172+0000", 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6209999999999993e-06 2020-03-09T00:25:51.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.632 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789181+0000", 2020-03-09T00:25:51.632 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4939999999999997e-06 2020-03-09T00:25:51.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.632 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789185+0000", 2020-03-09T00:25:51.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5793e-05 2020-03-09T00:25:51.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.633 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.633 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789231+0000", 2020-03-09T00:25:51.633 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7570000000000002e-06 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for peered", 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:57.789235+0000", 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.99423119699999996 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.634 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783466+0000", 2020-03-09T00:25:51.635 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000227851 2020-03-09T00:25:51.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.635 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:51.635 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.783694+0000", 2020-03-09T00:25:51.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056076839999999999 2020-03-09T00:25:51.636 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.636 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.636 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.789302+0000", 2020-03-09T00:25:51.636 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0001726259999999 2020-03-09T00:25:51.637 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.637 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.637 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.638 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.638 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.638 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 53/52 ECSubWriteReply(tid=1, last_complete=34'2, committed=1, applied=1))", 2020-03-09T00:25:51.638 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:51.638 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 48.823710419999998, 2020-03-09T00:25:51.638 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215, 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.639 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.640 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793298+0000", 2020-03-09T00:25:51.640 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9154999999999999e-05 2020-03-09T00:25:51.640 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.640 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.640 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793357+0000", 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4100000000000008e-06 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793366+0000", 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3783e-05 2020-03-09T00:25:51.641 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.642 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.642 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793380+0000", 2020-03-09T00:25:51.642 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6469999999999998e-06 2020-03-09T00:25:51.642 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.642 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.642 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.793387+0000", 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012002081 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805389+0000", 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0139000000000001e-05 2020-03-09T00:25:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.644 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.644 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.644 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.805409+0000", 2020-03-09T00:25:51.644 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012111215 2020-03-09T00:25:51.644 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.644 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.644 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 53/52 ECSubWrite(tid=47, reqid=client.4751.0:355, at_version=53'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 48.765825675999999, 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252, 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.645 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.646 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.647 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851183+0000", 2020-03-09T00:25:51.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00081411500000000002 2020-03-09T00:25:51.647 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.647 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.647 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.647 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851997+0000", 2020-03-09T00:25:51.647 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0162999999999999e-05 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852027+0000", 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7360999999999999e-05 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.648 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852045+0000", 2020-03-09T00:25:51.649 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2069000000000001e-05 2020-03-09T00:25:51.649 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.649 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.649 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.649 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852057+0000", 2020-03-09T00:25:51.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000131192 2020-03-09T00:25:51.650 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.650 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.650 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.650 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852188+0000", 2020-03-09T00:25:51.650 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.517e-05 2020-03-09T00:25:51.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.651 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.651 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.852203+0000", 2020-03-09T00:25:51.651 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099256910000000004 2020-03-09T00:25:51.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.651 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.652 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.652 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862129+0000", 2020-03-09T00:25:51.652 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6758999999999997e-05 2020-03-09T00:25:51.652 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.652 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.652 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.653 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.862176+0000", 2020-03-09T00:25:51.654 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01099252 2020-03-09T00:25:51.654 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.654 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.654 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.654 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.654 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.655 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=159, reqid=osd.12.0:159, at_version=59'41, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:51.655 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:51.655 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.796694318999997, 2020-03-09T00:25:51.656 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001, 2020-03-09T00:25:51.656 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.656 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.656 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.657 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.657 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:51.657 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.658 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.658 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820314+0000", 2020-03-09T00:25:51.658 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1613999999999997e-05 2020-03-09T00:25:51.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820376+0000", 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6792e-05 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820403+0000", 2020-03-09T00:25:51.659 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4728e-05 2020-03-09T00:25:51.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.660 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.660 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.660 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820417+0000", 2020-03-09T00:25:51.660 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7910000000000005e-06 2020-03-09T00:25:51.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.660 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.661 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.661 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820425+0000", 2020-03-09T00:25:51.661 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010933589999999999 2020-03-09T00:25:51.661 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.661 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.661 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.661 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821519+0000", 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3885e-05 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821533+0000", 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011394939999999999 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.662 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.663 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.663 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832927+0000", 2020-03-09T00:25:51.663 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.6371e-05 2020-03-09T00:25:51.663 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.663 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.663 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832974+0000", 2020-03-09T00:25:51.664 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012659480000000001 2020-03-09T00:25:51.664 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.664 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.664 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=5, reqid=client.4751.0:485, at_version=59'26, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.796052383000003, 2020-03-09T00:25:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083, 2020-03-09T00:25:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.820956+0000", 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4487000000000002e-05 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821011+0000", 2020-03-09T00:25:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9616e-05 2020-03-09T00:25:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821030+0000", 2020-03-09T00:25:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4766000000000001e-05 2020-03-09T00:25:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821045+0000", 2020-03-09T00:25:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.789e-06 2020-03-09T00:25:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821051+0000", 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010220629999999999 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822073+0000", 2020-03-09T00:25:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1511999999999999e-05 2020-03-09T00:25:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.822085+0000", 2020-03-09T00:25:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010894232 2020-03-09T00:25:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.832979+0000", 2020-03-09T00:25:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3618000000000001e-05 2020-03-09T00:25:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833022+0000", 2020-03-09T00:25:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012066083 2020-03-09T00:25:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=6, reqid=client.4751.0:486, at_version=59'27, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.795907501000002, 2020-03-09T00:25:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978, 2020-03-09T00:25:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821101+0000", 2020-03-09T00:25:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4271000000000003e-05 2020-03-09T00:25:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821155+0000", 2020-03-09T00:25:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8627e-05 2020-03-09T00:25:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821174+0000", 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4825e-05 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821189+0000", 2020-03-09T00:25:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4070000000000001e-06 2020-03-09T00:25:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821194+0000", 2020-03-09T00:25:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039590839999999999 2020-03-09T00:25:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825153+0000", 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3074999999999999e-05 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.825166+0000", 2020-03-09T00:25:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078593580000000003 2020-03-09T00:25:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833026+0000", 2020-03-09T00:25:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2330999999999999e-05 2020-03-09T00:25:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833058+0000", 2020-03-09T00:25:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011956978 2020-03-09T00:25:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=161, reqid=osd.12.0:161, at_version=59'43, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.795702478999999, 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525, 2020-03-09T00:25:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821306+0000", 2020-03-09T00:25:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6616999999999997e-05 2020-03-09T00:25:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821363+0000", 2020-03-09T00:25:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1061e-05 2020-03-09T00:25:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821384+0000", 2020-03-09T00:25:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5315000000000002e-05 2020-03-09T00:25:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821399+0000", 2020-03-09T00:25:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0929999999999999e-06 2020-03-09T00:25:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.821405+0000", 2020-03-09T00:25:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003024559 2020-03-09T00:25:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824430+0000", 2020-03-09T00:25:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2744e-05 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.824443+0000", 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010025054 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834468+0000", 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.0082e-05 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.834508+0000", 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013201525 2020-03-09T00:25:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.8s2 59/54 ECSubWrite(tid=7, reqid=client.4751.0:487, at_version=59'28, trim_to=0'0, roll_forward_to=57'24))", 2020-03-09T00:25:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.777359887999999, 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916, 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839649+0000", 2020-03-09T00:25:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.7596999999999998e-05 2020-03-09T00:25:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839706+0000", 2020-03-09T00:25:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5558000000000002e-05 2020-03-09T00:25:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839732+0000", 2020-03-09T00:25:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0251999999999999e-05 2020-03-09T00:25:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839752+0000", 2020-03-09T00:25:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0780000000000002e-06 2020-03-09T00:25:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839761+0000", 2020-03-09T00:25:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010019800000000001 2020-03-09T00:25:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839861+0000", 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9032e-05 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839881+0000", 2020-03-09T00:25:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014066339000000001 2020-03-09T00:25:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.853947+0000", 2020-03-09T00:25:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000210862 2020-03-09T00:25:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.854158+0000", 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014508916 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.0s5 59/54 ECSubWrite(tid=164, reqid=client.4751.0:490, at_version=59'45, trim_to=0'0, roll_forward_to=58'39))", 2020-03-09T00:25:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.777141456999999, 2020-03-09T00:25:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001, 2020-03-09T00:25:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.839867+0000", 2020-03-09T00:25:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000195997 2020-03-09T00:25:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840063+0000", 2020-03-09T00:25:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7005e-05 2020-03-09T00:25:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840090+0000", 2020-03-09T00:25:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6943999999999999e-05 2020-03-09T00:25:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840107+0000", 2020-03-09T00:25:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7489999999999998e-06 2020-03-09T00:25:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840115+0000", 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019725999999999999 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840312+0000", 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4409e-05 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.840327+0000", 2020-03-09T00:25:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018701885000000001 2020-03-09T00:25:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859028+0000", 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7057e-05 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859076+0000", 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019208306000000001 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.773632677000002, 2020-03-09T00:25:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002, 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843376+0000", 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012568360000000001 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844633+0000", 2020-03-09T00:25:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.0247999999999999e-05 2020-03-09T00:25:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844663+0000", 2020-03-09T00:25:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6184e-05 2020-03-09T00:25:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844679+0000", 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.8560000000000001e-06 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844686+0000", 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8074e-05 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844744+0000", 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9119000000000001e-05 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844763+0000", 2020-03-09T00:25:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014486196999999999 2020-03-09T00:25:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859250+0000", 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000115911 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.859365+0000", 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015989425000000002 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 59/54 ECSubWrite(tid=90, reqid=client.4751.0:499, at_version=59'56, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.767926905000003, 2020-03-09T00:25:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999, 2020-03-09T00:25:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849082+0000", 2020-03-09T00:25:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050415699999999998 2020-03-09T00:25:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849586+0000", 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9476e-05 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849615+0000", 2020-03-09T00:25:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.8382000000000001e-05 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849634+0000", 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0439999999999996e-06 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.849642+0000", 2020-03-09T00:25:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018423140000000001 2020-03-09T00:25:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851484+0000", 2020-03-09T00:25:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3304e-05 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.851497+0000", 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0097988120000000005 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861296+0000", 2020-03-09T00:25:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1790999999999999e-05 2020-03-09T00:25:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861348+0000", 2020-03-09T00:25:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012266279999999999 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:607 3.3s0 3:c2119ac7:::smithi10428818-4:head [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:51.724 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 32.815458714999998, 2020-03-09T00:25:51.725 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011089972999999999, 2020-03-09T00:25:51.725 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.725 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:51.725 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:51.725 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:51.725 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:51.725 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 607 2020-03-09T00:25:51.726 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.726 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.726 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.726 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.726 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:51.726 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801550+0000", 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7959e-05 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.727 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801588+0000", 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.676e-06 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801592+0000", 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3799999999999999e-06 2020-03-09T00:25:51.728 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801598+0000", 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5610000000000004e-06 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.729 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801605+0000", 2020-03-09T00:25:51.730 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015221259999999999 2020-03-09T00:25:51.730 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.730 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.730 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.730 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.803127+0000", 2020-03-09T00:25:51.730 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012211399999999999 2020-03-09T00:25:51.730 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.803249+0000", 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0093911570000000007 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.731 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.812640+0000", 2020-03-09T00:25:51.732 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011089972999999999 2020-03-09T00:25:51.732 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.732 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.732 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.732 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.732 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.732 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:608 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 32.815373479000002, 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013652041, 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:51.733 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 608 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.734 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.735 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.735 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.735 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801635+0000", 2020-03-09T00:25:51.735 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5256999999999997e-05 2020-03-09T00:25:51.735 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.735 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.735 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801680+0000", 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3590000000000002e-06 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801688+0000", 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1552999999999999e-05 2020-03-09T00:25:51.736 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801699+0000", 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.3519999999999997e-06 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.737 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801705+0000", 2020-03-09T00:25:51.738 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003598459 2020-03-09T00:25:51.738 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.738 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.738 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.738 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.805303+0000", 2020-03-09T00:25:51.738 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000173425 2020-03-09T00:25:51.738 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.805477+0000", 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098106359999999993 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.739 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.815287+0000", 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013652041 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:609 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:51.740 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:51.741 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 32.815253890999998, 2020-03-09T00:25:51.741 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013941073999999999, 2020-03-09T00:25:51.741 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.741 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:51.741 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:51.741 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:51.741 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 609 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.742 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.743 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.743 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.743 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801755+0000", 2020-03-09T00:25:51.743 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0500000000000006e-05 2020-03-09T00:25:51.743 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.743 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.743 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801825+0000", 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.614e-06 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801831+0000", 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5819999999999998e-06 2020-03-09T00:25:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.745 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.745 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.745 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801837+0000", 2020-03-09T00:25:51.745 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.619e-06 2020-03-09T00:25:51.745 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.745 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.745 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801841+0000", 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042523969999999998 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806094+0000", 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011631799999999999 2020-03-09T00:25:51.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806210+0000", 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094860440000000008 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.815696+0000", 2020-03-09T00:25:51.748 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013941073999999999 2020-03-09T00:25:51.748 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.748 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.748 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.748 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.748 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.748 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:610 3.3s0 3:c7fddd94:::smithi10428818-1:head [read 0~32768,checksum] snapc 0=[] ondisk+read+known_if_redirected e69)", 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 32.815116473000003, 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014287645, 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:51.749 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:51.750 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 610 2020-03-09T00:25:51.750 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.750 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.750 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.750 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.750 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:51.750 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801892+0000", 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5413000000000005e-05 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.751 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.752 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801968+0000", 2020-03-09T00:25:51.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4571e-05 2020-03-09T00:25:51.752 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.752 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.752 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.752 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801982+0000", 2020-03-09T00:25:51.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3568e-05 2020-03-09T00:25:51.753 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.753 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.753 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.753 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.801996+0000", 2020-03-09T00:25:51.753 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4940000000000003e-06 2020-03-09T00:25:51.753 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.753 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.802001+0000", 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046307759999999996 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806632+0000", 2020-03-09T00:25:51.754 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011677599999999999 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.806749+0000", 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0094310469999999997 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.755 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.756 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:18.816180+0000", 2020-03-09T00:25:51.756 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014287645 2020-03-09T00:25:51.756 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.756 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.756 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.756 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.757 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.4s6 75/67 ECSubWrite(tid=93, reqid=client.4751.0:637, at_version=75'8, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.757 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:51.757 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 26.807241312999999, 2020-03-09T00:25:51.757 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021185215, 2020-03-09T00:25:51.757 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.757 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.757 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.809767+0000", 2020-03-09T00:25:51.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021834200000000002 2020-03-09T00:25:51.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.759 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.759 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811951+0000", 2020-03-09T00:25:51.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8116999999999999e-05 2020-03-09T00:25:51.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811979+0000", 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3502e-05 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811992+0000", 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9779999999999999e-06 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.811999+0000", 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6839999999999998e-05 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.762 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.812056+0000", 2020-03-09T00:25:51.762 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5362e-05 2020-03-09T00:25:51.762 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.762 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.762 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.812072+0000", 2020-03-09T00:25:51.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01883961 2020-03-09T00:25:51.765 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.765 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.830911+0000", 2020-03-09T00:25:51.766 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.1386e-05 2020-03-09T00:25:51.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.766 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.766 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.766 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:24.830953+0000", 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021185215 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.6s1 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:51.767 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.820270+0000", 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 21.796738194, 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013028451, 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.820270+0000", 2020-03-09T00:25:51.769 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.769 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.769 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.820270+0000", 2020-03-09T00:25:51.769 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001056408 2020-03-09T00:25:51.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821327+0000", 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.2941000000000002e-05 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821360+0000", 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4844999999999999e-05 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821375+0000", 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9319999999999999e-06 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.772 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821382+0000", 2020-03-09T00:25:51.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000103878 2020-03-09T00:25:51.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.772 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821485+0000", 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2746e-05 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.821498+0000", 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011737274000000001 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.833236+0000", 2020-03-09T00:25:51.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3427000000000001e-05 2020-03-09T00:25:51.774 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.833299+0000", 2020-03-09T00:25:51.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013028451 2020-03-09T00:25:51.775 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.775 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.775 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.775 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.884724+0000", 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 16.732284899, 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019060613000000001, 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884724+0000", 2020-03-09T00:25:51.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.777 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.777 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884724+0000", 2020-03-09T00:25:51.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0022766119999999999 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887000+0000", 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1268999999999999e-05 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.778 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887022+0000", 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2214e-05 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887034+0000", 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5749999999999997e-06 2020-03-09T00:25:51.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887040+0000", 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001634417 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.888675+0000", 2020-03-09T00:25:51.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5466999999999999e-05 2020-03-09T00:25:51.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:51.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.888690+0000", 2020-03-09T00:25:51.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015039879000000001 2020-03-09T00:25:51.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.903730+0000", 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.418e-05 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.903784+0000", 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019060613000000001 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:942 3.3s0 3:c2119ac7:::smithi10428818-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e95)", 2020-03-09T00:25:51.783 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:44.897955+0000", 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 6.7190533820000002, 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011257730000000001, 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:51.784 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 942 2020-03-09T00:25:51.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.785 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:51.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:51.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.897955+0000", 2020-03-09T00:25:51.785 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:51.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:51.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.897955+0000", 2020-03-09T00:25:51.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.9678000000000001e-05 2020-03-09T00:25:51.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.898005+0000", 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2949999999999994e-06 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.898013+0000", 2020-03-09T00:25:51.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5231e-05 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.898029+0000", 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2369999999999997e-06 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:51.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.898035+0000", 2020-03-09T00:25:51.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045111209999999999 2020-03-09T00:25:51.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.789 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.789 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.902546+0000", 2020-03-09T00:25:51.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1442999999999998e-05 2020-03-09T00:25:51.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.790 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:51.790 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.902637+0000", 2020-03-09T00:25:51.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0051003089999999999 2020-03-09T00:25:51.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.791 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:51.791 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.907738+0000", 2020-03-09T00:25:51.791 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0616000000000002e-05 2020-03-09T00:25:51.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.907818+0000", 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013948000000000001 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:44.909213+0000", 2020-03-09T00:25:51.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011257730000000001 2020-03-09T00:25:51.793 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.793 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.793 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.793 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:51.793 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:51.793 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:51.795 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-03-09T00:25:51.901 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is set 2020-03-09T00:25:51.925 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:51.925 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:51.926 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:51.926 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:51.934 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-03-09T00:25:52.049 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:52.049 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:52.049 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:52.049 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:52.049 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:52.063 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-03-09T00:25:52.180 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:52.180 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:52.180 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:52.180 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:52.180 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.181 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:207 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 5] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:52.181 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:52.181 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 64.320202159999994, 2020-03-09T00:25:52.181 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001, 2020-03-09T00:25:52.181 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.181 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.181 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.182 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.182 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 207 2020-03-09T00:25:52.182 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.182 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.182 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:52.183 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.183 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.183 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.183 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.183 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856918+0000", 2020-03-09T00:25:52.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7176999999999996e-05 2020-03-09T00:25:52.184 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.184 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.184 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.184 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.856995+0000", 2020-03-09T00:25:52.184 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7161999999999999e-05 2020-03-09T00:25:52.185 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.185 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.185 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.185 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857013+0000", 2020-03-09T00:25:52.185 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2879e-05 2020-03-09T00:25:52.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.186 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.186 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857026+0000", 2020-03-09T00:25:52.186 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6669999999999996e-06 2020-03-09T00:25:52.186 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.186 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857033+0000", 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6767999999999997e-05 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857080+0000", 2020-03-09T00:25:52.187 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4165999999999994e-05 2020-03-09T00:25:52.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.188 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.188 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857154+0000", 2020-03-09T00:25:52.188 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0068311889999999997 2020-03-09T00:25:52.188 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.188 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.863985+0000", 2020-03-09T00:25:52.189 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041278249999999999 2020-03-09T00:25:52.189 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.189 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.189 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.189 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868113+0000", 2020-03-09T00:25:52.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001971132 2020-03-09T00:25:52.190 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.190 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.190 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.190 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870084+0000", 2020-03-09T00:25:52.190 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000113266 2020-03-09T00:25:52.191 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.191 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.191 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.191 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.870198+0000", 2020-03-09T00:25:52.191 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013279231000000001 2020-03-09T00:25:52.191 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.192 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.192 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.192 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.192 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.192 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:208 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 64.320032413999996, 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001, 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 208 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.194 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.195 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.195 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857088+0000", 2020-03-09T00:25:52.195 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5717000000000003e-05 2020-03-09T00:25:52.195 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.195 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857134+0000", 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3659999999999997e-06 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857140+0000", 2020-03-09T00:25:52.196 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2546e-05 2020-03-09T00:25:52.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.197 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.197 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857153+0000", 2020-03-09T00:25:52.197 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7190000000000001e-06 2020-03-09T00:25:52.197 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.197 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857157+0000", 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000134509 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857292+0000", 2020-03-09T00:25:52.198 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5519999999999999e-05 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.857327+0000", 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010666132 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.199 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.867994+0000", 2020-03-09T00:25:52.200 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110682 2020-03-09T00:25:52.200 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.200 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.200 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:52.200 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.868104+0000", 2020-03-09T00:25:52.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00321515 2020-03-09T00:25:52.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.201 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.201 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.201 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871319+0000", 2020-03-09T00:25:52.201 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011457 2020-03-09T00:25:52.201 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.202 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871434+0000", 2020-03-09T00:25:52.202 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013121699999999999 2020-03-09T00:25:52.202 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.202 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.202 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.871565+0000", 2020-03-09T00:25:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014477128000000001 2020-03-09T00:25:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.203 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 58.326837597999997, 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998, 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.204 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.205 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:52.205 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.205 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.205 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.205 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850283+0000", 2020-03-09T00:25:52.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00018406799999999999 2020-03-09T00:25:52.206 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.206 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.206 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.206 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850467+0000", 2020-03-09T00:25:52.206 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2490999999999999e-05 2020-03-09T00:25:52.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.207 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.207 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850489+0000", 2020-03-09T00:25:52.207 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6807999999999999e-05 2020-03-09T00:25:52.207 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.207 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:52.208 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9419999999999995e-06 2020-03-09T00:25:52.208 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.208 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.208 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.208 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.850514+0000", 2020-03-09T00:25:52.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00088484100000000003 2020-03-09T00:25:52.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.209 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.209 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.209 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851399+0000", 2020-03-09T00:25:52.209 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2279999999999999e-05 2020-03-09T00:25:52.209 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.210 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.851411+0000", 2020-03-09T00:25:52.210 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0083268249999999995 2020-03-09T00:25:52.210 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.210 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.210 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859738+0000", 2020-03-09T00:25:52.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2222999999999998e-05 2020-03-09T00:25:52.211 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.211 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.211 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.211 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:53.859790+0000", 2020-03-09T00:25:52.211 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0095074779999999998 2020-03-09T00:25:52.212 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.212 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.212 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.212 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.212 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.es6 49/32 ECSubWrite(tid=44, reqid=client.4751.0:289, at_version=49'22, trim_to=0'0, roll_forward_to=48'20))", 2020-03-09T00:25:52.212 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:52.213 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 53.359722910000002, 2020-03-09T00:25:52.213 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001, 2020-03-09T00:25:52.213 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.213 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.213 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.213 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.817398+0000", 2020-03-09T00:25:52.214 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00069548899999999998 2020-03-09T00:25:52.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.215 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.215 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818093+0000", 2020-03-09T00:25:52.215 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6591999999999999e-05 2020-03-09T00:25:52.215 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.215 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.216 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818120+0000", 2020-03-09T00:25:52.216 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5192e-05 2020-03-09T00:25:52.216 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.216 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.216 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818135+0000", 2020-03-09T00:25:52.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4989999999999997e-06 2020-03-09T00:25:52.217 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.217 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.217 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.217 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818142+0000", 2020-03-09T00:25:52.217 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4997000000000002e-05 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818197+0000", 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3164e-05 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.218 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.818210+0000", 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092824440000000008 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827493+0000", 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2645999999999997e-05 2020-03-09T00:25:52.219 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:58.827546+0000", 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.010148023000000001 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.220 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.221 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.221 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.2s2 54/48 ECSubWrite(tid=8, reqid=client.4751.0:376, at_version=54'17, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:52.221 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:52.221 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 48.367709759999997, 2020-03-09T00:25:52.221 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002, 2020-03-09T00:25:52.221 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.222 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809411+0000", 2020-03-09T00:25:52.223 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000509736 2020-03-09T00:25:52.223 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.223 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.223 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.223 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809920+0000", 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8026999999999999e-05 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809938+0000", 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5829e-05 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.224 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809954+0000", 2020-03-09T00:25:52.225 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8799999999999997e-06 2020-03-09T00:25:52.225 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.225 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.225 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.225 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.809963+0000", 2020-03-09T00:25:52.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3013000000000003e-05 2020-03-09T00:25:52.226 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.226 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.226 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.226 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810006+0000", 2020-03-09T00:25:52.226 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2413e-05 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.810019+0000", 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0067421620000000003 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.227 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816761+0000", 2020-03-09T00:25:52.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3118999999999997e-05 2020-03-09T00:25:52.228 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.228 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.228 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.228 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:03.816814+0000", 2020-03-09T00:25:52.228 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0074031790000000002 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:507 3.as0 3:51058371:::smithi10428818-28:head [copy-from ver 54] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:52.229 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.272431011999998, 2020-03-09T00:25:52.230 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001, 2020-03-09T00:25:52.230 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.230 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.230 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 507 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.231 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.232 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904689+0000", 2020-03-09T00:25:52.232 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.4511999999999997e-05 2020-03-09T00:25:52.232 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.232 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.232 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904734+0000", 2020-03-09T00:25:52.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3980000000000004e-06 2020-03-09T00:25:52.233 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.233 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.233 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.233 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904740+0000", 2020-03-09T00:25:52.233 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4293e-05 2020-03-09T00:25:52.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.234 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.234 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904755+0000", 2020-03-09T00:25:52.234 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4449999999999996e-06 2020-03-09T00:25:52.234 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.234 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.235 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904763+0000", 2020-03-09T00:25:52.235 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0075999999999999e-05 2020-03-09T00:25:52.235 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.235 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.235 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904823+0000", 2020-03-09T00:25:52.236 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.8325999999999994e-05 2020-03-09T00:25:52.236 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.236 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.236 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.236 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904911+0000", 2020-03-09T00:25:52.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091150910000000005 2020-03-09T00:25:52.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.237 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.237 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.237 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914027+0000", 2020-03-09T00:25:52.237 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097858200000000006 2020-03-09T00:25:52.237 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923812+0000", 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072064440000000002 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.238 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931019+0000", 2020-03-09T00:25:52.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6091000000000001e-05 2020-03-09T00:25:52.239 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.239 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.239 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.239 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931085+0000", 2020-03-09T00:25:52.239 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.026395496000000001 2020-03-09T00:25:52.239 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:508 3.as0 3:51058371:::smithi10428818-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 43.272318943999998, 2020-03-09T00:25:52.240 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261, 2020-03-09T00:25:52.241 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.241 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:52.241 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.241 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.241 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.241 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 508 2020-03-09T00:25:52.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.242 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904802+0000", 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4413000000000002e-05 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904826+0000", 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1389999999999997e-06 2020-03-09T00:25:52.243 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.244 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904830+0000", 2020-03-09T00:25:52.244 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3339999999999996e-06 2020-03-09T00:25:52.244 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.244 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.244 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904837+0000", 2020-03-09T00:25:52.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8739999999999999e-06 2020-03-09T00:25:52.245 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.245 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.245 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.245 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.904840+0000", 2020-03-09T00:25:52.245 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000222303 2020-03-09T00:25:52.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.246 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.246 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.246 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905063+0000", 2020-03-09T00:25:52.246 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5607999999999999e-05 2020-03-09T00:25:52.246 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:52.247 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.905128+0000", 2020-03-09T00:25:52.247 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018331841000000001 2020-03-09T00:25:52.247 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.247 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.247 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923460+0000", 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012596800000000001 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.923586+0000", 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0084087399999999996 2020-03-09T00:25:52.248 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.249 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.931995+0000", 2020-03-09T00:25:52.249 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010665 2020-03-09T00:25:52.249 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.249 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.249 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932101+0000", 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012239099999999999 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.932224+0000", 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.027422261 2020-03-09T00:25:52.250 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:598 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.388583003999997, 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014064482999999999, 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 598 2020-03-09T00:25:52.252 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.253 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788537+0000", 2020-03-09T00:25:52.254 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4369999999999999e-05 2020-03-09T00:25:52.254 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.254 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.254 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.254 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788562+0000", 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8599999999999998e-06 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788568+0000", 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.5980000000000003e-06 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.255 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788577+0000", 2020-03-09T00:25:52.256 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.955e-06 2020-03-09T00:25:52.256 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.256 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.256 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.256 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788582+0000", 2020-03-09T00:25:52.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5593000000000002e-05 2020-03-09T00:25:52.257 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.257 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.257 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.257 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788628+0000", 2020-03-09T00:25:52.257 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3660000000000003e-06 2020-03-09T00:25:52.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.258 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.258 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "op must wait for map", 2020-03-09T00:25:52.258 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788635+0000", 2020-03-09T00:25:52.258 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0019670759999999999 2020-03-09T00:25:52.258 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.790602+0000", 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0575999999999995e-05 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.259 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.790693+0000", 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00085902600000000004 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.791552+0000", 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0052594449999999997 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.260 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796811+0000", 2020-03-09T00:25:52.261 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056778230000000002 2020-03-09T00:25:52.261 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.261 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.261 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.261 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802489+0000", 2020-03-09T00:25:52.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011279499999999999 2020-03-09T00:25:52.262 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.262 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.262 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.262 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802602+0000", 2020-03-09T00:25:52.262 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014064482999999999 2020-03-09T00:25:52.262 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.263 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.263 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.263 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.263 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.263 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:599 3.as0 3:51058371:::smithi10428818-28:head [rollback 1a] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:52.263 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:52.263 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.388494583000004, 2020-03-09T00:25:52.264 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014184085000000001, 2020-03-09T00:25:52.264 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.264 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.264 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.264 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 599 2020-03-09T00:25:52.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.265 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.265 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.265 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.265 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:52.265 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.265 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788626+0000", 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8149999999999998e-05 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.266 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788674+0000", 2020-03-09T00:25:52.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7279999999999998e-06 2020-03-09T00:25:52.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.267 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.267 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.267 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788681+0000", 2020-03-09T00:25:52.267 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.347e-05 2020-03-09T00:25:52.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.268 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.268 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788694+0000", 2020-03-09T00:25:52.268 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8910000000000003e-06 2020-03-09T00:25:52.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.268 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788699+0000", 2020-03-09T00:25:52.269 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.2481999999999998e-05 2020-03-09T00:25:52.269 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.269 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.269 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.269 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788732+0000", 2020-03-09T00:25:52.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4740000000000002e-06 2020-03-09T00:25:52.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.270 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.270 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:52.270 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788738+0000", 2020-03-09T00:25:52.270 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0044600949999999999 2020-03-09T00:25:52.270 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.271 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.271 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.793198+0000", 2020-03-09T00:25:52.271 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000122469 2020-03-09T00:25:52.271 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.271 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.793321+0000", 2020-03-09T00:25:52.272 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00081692400000000001 2020-03-09T00:25:52.272 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.272 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.272 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.272 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794138+0000", 2020-03-09T00:25:52.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0026755360000000001 2020-03-09T00:25:52.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.273 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.273 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.273 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796813+0000", 2020-03-09T00:25:52.273 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058884480000000001 2020-03-09T00:25:52.273 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.274 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802702+0000", 2020-03-09T00:25:52.274 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000108418 2020-03-09T00:25:52.274 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.274 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.274 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.802810+0000", 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014184085000000001 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.275 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:600 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 1f=[1f,1a] ondisk+write+known_if_redirected e69)", 2020-03-09T00:25:52.276 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:52.276 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 33.388371085000003, 2020-03-09T00:25:52.276 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016073481000000001, 2020-03-09T00:25:52.276 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.276 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.276 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.276 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.277 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.277 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 600 2020-03-09T00:25:52.277 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.277 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.277 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.277 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:52.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.278 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.278 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.278 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.278 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788749+0000", 2020-03-09T00:25:52.278 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.3720999999999998e-05 2020-03-09T00:25:52.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.279 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.279 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788783+0000", 2020-03-09T00:25:52.279 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.5499999999999999e-06 2020-03-09T00:25:52.279 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.279 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788787+0000", 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0769e-05 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788797+0000", 2020-03-09T00:25:52.280 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.157e-06 2020-03-09T00:25:52.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.281 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.281 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788801+0000", 2020-03-09T00:25:52.281 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1471000000000002e-05 2020-03-09T00:25:52.281 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.281 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788822+0000", 2020-03-09T00:25:52.282 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3599999999999998e-06 2020-03-09T00:25:52.282 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.282 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.282 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:52.282 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.788826+0000", 2020-03-09T00:25:52.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066200779999999997 2020-03-09T00:25:52.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.283 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.283 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.283 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.795446+0000", 2020-03-09T00:25:52.283 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013750200000000001 2020-03-09T00:25:52.283 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.284 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.795584+0000", 2020-03-09T00:25:52.284 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00096939200000000002 2020-03-09T00:25:52.284 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.284 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.284 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796553+0000", 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001230536 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.797784+0000", 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0069359829999999997 2020-03-09T00:25:52.285 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.286 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.804720+0000", 2020-03-09T00:25:52.286 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010296199999999999 2020-03-09T00:25:52.286 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.286 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.286 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.287 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.804823+0000", 2020-03-09T00:25:52.287 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.016073481000000001 2020-03-09T00:25:52.287 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.287 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.287 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.287 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.288 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.288 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:642 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:52.288 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:52.288 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.391766400000002, 2020-03-09T00:25:52.288 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014044918999999999, 2020-03-09T00:25:52.288 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.288 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.289 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.289 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.289 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.289 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 642 2020-03-09T00:25:52.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.289 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.289 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.290 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.290 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:52.290 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.290 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.290 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.291 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785354+0000", 2020-03-09T00:25:52.291 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0393999999999999e-05 2020-03-09T00:25:52.291 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.291 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.291 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.292 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785404+0000", 2020-03-09T00:25:52.292 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0550000000000005e-06 2020-03-09T00:25:52.292 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.292 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.292 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785414+0000", 2020-03-09T00:25:52.293 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6269999999999998e-05 2020-03-09T00:25:52.293 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.293 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.293 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.293 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785430+0000", 2020-03-09T00:25:52.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3169999999999998e-06 2020-03-09T00:25:52.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.294 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.294 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.294 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785436+0000", 2020-03-09T00:25:52.294 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00041141700000000002 2020-03-09T00:25:52.294 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785848+0000", 2020-03-09T00:25:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011097799999999999 2020-03-09T00:25:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.295 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785958+0000", 2020-03-09T00:25:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001069469 2020-03-09T00:25:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787028+0000", 2020-03-09T00:25:52.296 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082925840000000004 2020-03-09T00:25:52.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.297 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.297 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.297 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.795321+0000", 2020-03-09T00:25:52.297 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039837429999999997 2020-03-09T00:25:52.297 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.298 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799304+0000", 2020-03-09T00:25:52.298 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.4691999999999994e-05 2020-03-09T00:25:52.298 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.298 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.298 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.299 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799399+0000", 2020-03-09T00:25:52.299 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014044918999999999 2020-03-09T00:25:52.299 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.299 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.299 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.299 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.299 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.300 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:643 3.as0 3:51058371:::smithi10428818-28:head [rollback 24] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:52.300 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:52.300 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.391624243999999, 2020-03-09T00:25:52.300 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014330799999999999, 2020-03-09T00:25:52.300 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.300 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.300 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.301 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.301 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.301 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 643 2020-03-09T00:25:52.301 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.301 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.301 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.301 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:52.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.302 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.302 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.302 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.302 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785496+0000", 2020-03-09T00:25:52.302 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6378999999999998e-05 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785543+0000", 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4949999999999996e-06 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.303 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785549+0000", 2020-03-09T00:25:52.304 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4008e-05 2020-03-09T00:25:52.304 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.304 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.304 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.304 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785563+0000", 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8459999999999999e-06 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785568+0000", 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0022776580000000001 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787846+0000", 2020-03-09T00:25:52.306 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00011612699999999999 2020-03-09T00:25:52.306 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.306 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.306 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.306 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.787962+0000", 2020-03-09T00:25:52.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.002561486 2020-03-09T00:25:52.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.307 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.307 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.307 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.790523+0000", 2020-03-09T00:25:52.307 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0059343010000000003 2020-03-09T00:25:52.307 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.308 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.308 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796458+0000", 2020-03-09T00:25:52.308 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003260359 2020-03-09T00:25:52.308 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.308 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.309 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799718+0000", 2020-03-09T00:25:52.309 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000109141 2020-03-09T00:25:52.309 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.309 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.309 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.310 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799827+0000", 2020-03-09T00:25:52.310 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014330799999999999 2020-03-09T00:25:52.310 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.310 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.310 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.310 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.310 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.311 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:644 3.as0 3:51058371:::smithi10428818-28:head [append 0~0] snapc 26=[24,23,22,1a] ondisk+write+known_if_redirected e75)", 2020-03-09T00:25:52.311 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:52.311 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 27.391504489999999, 2020-03-09T00:25:52.311 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014401074, 2020-03-09T00:25:52.311 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.311 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.311 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.312 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.312 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.312 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 644 2020-03-09T00:25:52.312 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.312 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.312 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.312 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:52.313 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.313 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.313 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.313 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.313 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785616+0000", 2020-03-09T00:25:52.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8556000000000001e-05 2020-03-09T00:25:52.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.314 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.314 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.314 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785665+0000", 2020-03-09T00:25:52.314 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.1480000000000003e-06 2020-03-09T00:25:52.314 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785671+0000", 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1841999999999999e-05 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.315 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785683+0000", 2020-03-09T00:25:52.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.907e-06 2020-03-09T00:25:52.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.316 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.316 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.316 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.785686+0000", 2020-03-09T00:25:52.316 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0060596110000000003 2020-03-09T00:25:52.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.317 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.791746+0000", 2020-03-09T00:25:52.317 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013271499999999999 2020-03-09T00:25:52.317 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.317 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.317 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.318 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.791879+0000", 2020-03-09T00:25:52.318 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00068155099999999997 2020-03-09T00:25:52.318 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.318 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.318 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.792560+0000", 2020-03-09T00:25:52.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0039004560000000001 2020-03-09T00:25:52.319 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.319 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.319 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.319 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.796461+0000", 2020-03-09T00:25:52.319 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0034719059999999999 2020-03-09T00:25:52.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.320 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.320 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.799933+0000", 2020-03-09T00:25:52.320 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4382000000000006e-05 2020-03-09T00:25:52.320 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.320 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.321 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.321 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:24.800017+0000", 2020-03-09T00:25:52.321 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014401074 2020-03-09T00:25:52.321 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.321 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.321 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.321 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.322 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.322 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:909 3.as0 3:51058371:::smithi10428818-28:head [append 0~458752] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:52.322 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.880547+0000", 2020-03-09T00:25:52.322 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.2965732039999995, 2020-03-09T00:25:52.322 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018015781000000002, 2020-03-09T00:25:52.322 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.322 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.323 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.323 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 909 2020-03-09T00:25:52.323 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.323 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.323 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880547+0000", 2020-03-09T00:25:52.324 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.324 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.324 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.324 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.324 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880547+0000", 2020-03-09T00:25:52.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00021156899999999999 2020-03-09T00:25:52.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.325 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.325 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.325 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880759+0000", 2020-03-09T00:25:52.325 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8810000000000001e-06 2020-03-09T00:25:52.325 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.326 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880765+0000", 2020-03-09T00:25:52.326 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2879999999999998e-06 2020-03-09T00:25:52.326 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.326 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.326 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880774+0000", 2020-03-09T00:25:52.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2379999999999999e-06 2020-03-09T00:25:52.327 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.327 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.327 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.327 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880778+0000", 2020-03-09T00:25:52.327 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.3570999999999999e-05 2020-03-09T00:25:52.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.328 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.328 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.328 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880832+0000", 2020-03-09T00:25:52.328 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.0734000000000002e-05 2020-03-09T00:25:52.328 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.880923+0000", 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012208919999999999 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.329 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882143+0000", 2020-03-09T00:25:52.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046134339999999996 2020-03-09T00:25:52.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.330 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.330 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.330 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.886757+0000", 2020-03-09T00:25:52.330 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011695929000000001 2020-03-09T00:25:52.330 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.331 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898453+0000", 2020-03-09T00:25:52.331 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110245 2020-03-09T00:25:52.331 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.331 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.331 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.332 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898563+0000", 2020-03-09T00:25:52.332 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.018015781000000002 2020-03-09T00:25:52.332 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.332 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.332 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.332 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.332 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.333 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:910 3.as0 3:51058371:::smithi10428818-28:head [append 0~475136] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:52.333 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.887513+0000", 2020-03-09T00:25:52.333 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.2896075979999999, 2020-03-09T00:25:52.333 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011280438, 2020-03-09T00:25:52.333 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.333 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.333 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.334 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.334 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 910 2020-03-09T00:25:52.334 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.334 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.334 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.334 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887513+0000", 2020-03-09T00:25:52.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.335 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.335 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.335 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.335 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887513+0000", 2020-03-09T00:25:52.335 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000323393 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887836+0000", 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.02e-06 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.336 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887845+0000", 2020-03-09T00:25:52.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4946000000000001e-05 2020-03-09T00:25:52.337 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.337 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.337 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.337 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887860+0000", 2020-03-09T00:25:52.337 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0669999999999997e-06 2020-03-09T00:25:52.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.338 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.338 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.338 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887866+0000", 2020-03-09T00:25:52.338 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5791999999999998e-05 2020-03-09T00:25:52.338 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.339 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.887912+0000", 2020-03-09T00:25:52.339 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000145005 2020-03-09T00:25:52.339 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.339 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.339 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.888057+0000", 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012473250000000001 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.889304+0000", 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0088728850000000005 2020-03-09T00:25:52.340 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.341 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898177+0000", 2020-03-09T00:25:52.341 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00052626699999999997 2020-03-09T00:25:52.341 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.341 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.341 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898704+0000", 2020-03-09T00:25:52.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.9738e-05 2020-03-09T00:25:52.342 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.342 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.342 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.342 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898793+0000", 2020-03-09T00:25:52.342 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011280438 2020-03-09T00:25:52.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.343 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.343 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.343 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.344 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.344 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:911 3.as0 3:51058371:::smithi10428818-28:head [append 0~655360] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:52.344 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.891986+0000", 2020-03-09T00:25:52.344 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.2851342209999999, 2020-03-09T00:25:52.344 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21223953100000001, 2020-03-09T00:25:52.344 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.344 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.345 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.345 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.345 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.345 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 911 2020-03-09T00:25:52.345 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.345 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.345 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.891986+0000", 2020-03-09T00:25:52.346 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.346 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.346 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.346 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.346 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.891986+0000", 2020-03-09T00:25:52.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00020587900000000001 2020-03-09T00:25:52.347 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.347 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.347 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.347 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.892192+0000", 2020-03-09T00:25:52.347 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4640000000000006e-06 2020-03-09T00:25:52.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.348 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.348 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.892201+0000", 2020-03-09T00:25:52.348 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7125e-05 2020-03-09T00:25:52.348 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.348 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.349 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.892218+0000", 2020-03-09T00:25:52.349 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.0839999999999992e-06 2020-03-09T00:25:52.349 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.349 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.349 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.892226+0000", 2020-03-09T00:25:52.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000713063 2020-03-09T00:25:52.350 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.350 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.350 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.350 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.892939+0000", 2020-03-09T00:25:52.350 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00014042699999999999 2020-03-09T00:25:52.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.351 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.351 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.893079+0000", 2020-03-09T00:25:52.351 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012312860000000001 2020-03-09T00:25:52.351 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.351 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894311+0000", 2020-03-09T00:25:52.352 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0091159610000000005 2020-03-09T00:25:52.352 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.352 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.352 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.352 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.903427+0000", 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20073924600000001 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104166+0000", 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9996e-05 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.353 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.354 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.354 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104226+0000", 2020-03-09T00:25:52.354 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.21223953100000001 2020-03-09T00:25:52.354 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.354 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.354 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.354 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.355 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.355 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:912 3.as0 3:51058371:::smithi10428818-28:head [append 0~671744] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:52.355 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.896874+0000", 2020-03-09T00:25:52.355 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.2802463409999998, 2020-03-09T00:25:52.355 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.207624379, 2020-03-09T00:25:52.355 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.355 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.356 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.356 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 912 2020-03-09T00:25:52.356 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.356 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.356 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.896874+0000", 2020-03-09T00:25:52.357 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.357 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.357 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.357 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.357 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.896874+0000", 2020-03-09T00:25:52.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00043218500000000002 2020-03-09T00:25:52.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.358 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.358 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.358 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897306+0000", 2020-03-09T00:25:52.358 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2072000000000001e-05 2020-03-09T00:25:52.358 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.359 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.359 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897318+0000", 2020-03-09T00:25:52.359 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5531000000000001e-05 2020-03-09T00:25:52.359 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.359 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.360 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897334+0000", 2020-03-09T00:25:52.360 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7240000000000003e-06 2020-03-09T00:25:52.360 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.360 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.360 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897341+0000", 2020-03-09T00:25:52.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0015619519999999999 2020-03-09T00:25:52.361 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.361 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.361 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.361 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898903+0000", 2020-03-09T00:25:52.361 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013719599999999999 2020-03-09T00:25:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899040+0000", 2020-03-09T00:25:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0012896050000000001 2020-03-09T00:25:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.362 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.900329+0000", 2020-03-09T00:25:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054226350000000003 2020-03-09T00:25:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.363 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.905752+0000", 2020-03-09T00:25:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19864865000000001 2020-03-09T00:25:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104401+0000", 2020-03-09T00:25:52.364 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.7829000000000001e-05 2020-03-09T00:25:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104498+0000", 2020-03-09T00:25:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.207624379 2020-03-09T00:25:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.365 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:913 3.as0 3:51058371:::smithi10428818-28:head [append 0~147456] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.897835+0000", 2020-03-09T00:25:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.2792851610000007, 2020-03-09T00:25:52.366 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20690554799999999, 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 913 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897835+0000", 2020-03-09T00:25:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897835+0000", 2020-03-09T00:25:52.369 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000150047 2020-03-09T00:25:52.369 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.369 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.369 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.369 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897985+0000", 2020-03-09T00:25:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8679999999999999e-06 2020-03-09T00:25:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.897993+0000", 2020-03-09T00:25:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4504999999999999e-05 2020-03-09T00:25:52.370 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898008+0000", 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0469999999999999e-06 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.371 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898014+0000", 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056303789999999996 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.903644+0000", 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000159025 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.372 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.903803+0000", 2020-03-09T00:25:52.373 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00090761300000000004 2020-03-09T00:25:52.373 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.373 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.373 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.373 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.904711+0000", 2020-03-09T00:25:52.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0021801849999999998 2020-03-09T00:25:52.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.374 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.374 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.374 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.906891+0000", 2020-03-09T00:25:52.374 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19776495099999999 2020-03-09T00:25:52.374 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.377 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.377 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.377 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104656+0000", 2020-03-09T00:25:52.378 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.4927999999999997e-05 2020-03-09T00:25:52.378 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.378 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.378 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.379 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104741+0000", 2020-03-09T00:25:52.379 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20690554799999999 2020-03-09T00:25:52.379 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.381 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.381 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.382 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.382 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.382 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:914 3.as0 3:51058371:::smithi10428818-28:head [setxattr _header (57) in=7b] snapc 3a=[3a,39,38] ondisk+write+known_if_redirected e94)", 2020-03-09T00:25:52.382 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.898069+0000", 2020-03-09T00:25:52.382 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.2790517309999991, 2020-03-09T00:25:52.382 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20688202, 2020-03-09T00:25:52.382 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.383 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:52.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.383 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.383 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 914 2020-03-09T00:25:52.383 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.383 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898069+0000", 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.384 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898069+0000", 2020-03-09T00:25:52.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9763000000000003e-05 2020-03-09T00:25:52.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.385 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.385 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.385 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898138+0000", 2020-03-09T00:25:52.385 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1699999999999999e-06 2020-03-09T00:25:52.385 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.386 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898143+0000", 2020-03-09T00:25:52.386 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.3309999999999998e-06 2020-03-09T00:25:52.386 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.386 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.386 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898150+0000", 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.304e-06 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898153+0000", 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0076328250000000002 2020-03-09T00:25:52.387 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.905786+0000", 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127436 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.388 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.905914+0000", 2020-03-09T00:25:52.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00071484300000000005 2020-03-09T00:25:52.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.389 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.389 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.389 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.906628+0000", 2020-03-09T00:25:52.389 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.001018281 2020-03-09T00:25:52.389 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.390 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.907647+0000", 2020-03-09T00:25:52.390 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.19719873800000001 2020-03-09T00:25:52.390 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.390 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.390 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "commit_sent", 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104845+0000", 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010532899999999999 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.104951+0000", 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.20688202 2020-03-09T00:25:52.391 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "osd_op(client.4751.0:915 3.as0 3:51058371:::smithi10428818-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e94)", 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.898190+0000", 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 8.278930012, 2020-03-09T00:25:52.392 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.209119367, 2020-03-09T00:25:52.393 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:52.393 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "started", 2020-03-09T00:25:52.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_info": { 2020-03-09T00:25:52.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client": "client.4751", 2020-03-09T00:25:52.393 INFO:teuthology.orchestra.run.smithi104.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:52.393 INFO:teuthology.orchestra.run.smithi104.stdout: "tid": 915 2020-03-09T00:25:52.393 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898190+0000", 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.394 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:52.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898190+0000", 2020-03-09T00:25:52.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9118e-05 2020-03-09T00:25:52.395 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.395 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.395 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:52.395 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898240+0000", 2020-03-09T00:25:52.395 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.438e-06 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898246+0000", 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4003e-05 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.396 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:52.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898260+0000", 2020-03-09T00:25:52.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6569999999999996e-06 2020-03-09T00:25:52.397 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.397 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.397 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.397 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.898265+0000", 2020-03-09T00:25:52.397 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0092045960000000007 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.907469+0000", 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012888099999999999 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.398 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.907598+0000", 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.197925398 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.105524+0000", 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000127634 2020-03-09T00:25:52.399 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "started", 2020-03-09T00:25:52.400 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.105651+0000", 2020-03-09T00:25:52.400 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016586420000000001 2020-03-09T00:25:52.400 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:52.400 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:52.400 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:44.107310+0000", 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.209119367 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:52.401 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:52.403 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-03-09T00:25:52.530 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:52.530 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:52.530 INFO:teuthology.orchestra.run.smithi139.stdout: "num_ops": 0 2020-03-09T00:25:52.530 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:52.540 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-03-09T00:25:52.655 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:52.656 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [], 2020-03-09T00:25:52.656 INFO:teuthology.orchestra.run.smithi139.stdout: "complaint_time": 30, 2020-03-09T00:25:52.656 INFO:teuthology.orchestra.run.smithi139.stdout: "num_blocked_ops": 0 2020-03-09T00:25:52.656 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:52.664 INFO:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-03-09T00:25:52.781 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2020-03-09T00:25:52.781 INFO:teuthology.orchestra.run.smithi139.stdout: "size": 20, 2020-03-09T00:25:52.781 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 600, 2020-03-09T00:25:52.782 INFO:teuthology.orchestra.run.smithi139.stdout: "ops": [ 2020-03-09T00:25:52.782 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.782 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 44/32 ECSubWrite(tid=36, reqid=client.4751.0:242, at_version=44'16, trim_to=0'0, roll_forward_to=38'14))", 2020-03-09T00:25:52.782 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:52.782 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 58.927274621000002, 2020-03-09T00:25:52.782 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999, 2020-03-09T00:25:52.782 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.783 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.783 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.783 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.783 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.783 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:52.783 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.783 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.785 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.785 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851380+0000", 2020-03-09T00:25:52.785 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017980700000000001 2020-03-09T00:25:52.785 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.785 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.786 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.786 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851560+0000", 2020-03-09T00:25:52.786 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.5363000000000001e-05 2020-03-09T00:25:52.786 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.786 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.786 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.786 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851585+0000", 2020-03-09T00:25:52.787 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4187000000000001e-05 2020-03-09T00:25:52.787 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.787 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.787 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.787 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851609+0000", 2020-03-09T00:25:52.787 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2890000000000003e-06 2020-03-09T00:25:52.787 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851618+0000", 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00036370100000000002 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.788 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851982+0000", 2020-03-09T00:25:52.789 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3288999999999999e-05 2020-03-09T00:25:52.789 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.789 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.789 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.789 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.851995+0000", 2020-03-09T00:25:52.789 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074675740000000003 2020-03-09T00:25:52.789 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.790 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.790 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859463+0000", 2020-03-09T00:25:52.790 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1719999999999999e-05 2020-03-09T00:25:52.790 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.790 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.790 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.859525+0000", 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081449299999999999 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.791 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=56, reqid=client.4751.0:254, at_version=44'23, trim_to=0'0, roll_forward_to=44'21))", 2020-03-09T00:25:52.792 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:52.792 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 58.914953613999998, 2020-03-09T00:25:52.792 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996, 2020-03-09T00:25:52.792 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.792 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.792 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.792 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.793 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.793 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:52.793 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.793 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.793 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.793 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.793 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863701+0000", 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00015903600000000001 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863860+0000", 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3975999999999999e-05 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.794 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863884+0000", 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.6139999999999998e-06 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863893+0000", 2020-03-09T00:25:52.795 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3190000000000002e-06 2020-03-09T00:25:52.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.796 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.796 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.863899+0000", 2020-03-09T00:25:52.796 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0039154710000000002 2020-03-09T00:25:52.796 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.796 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867814+0000", 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5379999999999998e-05 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.867830+0000", 2020-03-09T00:25:52.797 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001627606 2020-03-09T00:25:52.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.798 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.798 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.798 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869457+0000", 2020-03-09T00:25:52.798 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7565999999999998e-05 2020-03-09T00:25:52.798 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.798 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:53.869505+0000", 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058039679999999996 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.799 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=77, reqid=client.4751.0:301, at_version=49'12, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.947955790999998, 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001, 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.800 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.801 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.801 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:52.801 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.801 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.801 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.801 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.801 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.830699+0000", 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000394366 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831093+0000", 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6948999999999999e-05 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.802 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.803 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.803 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831120+0000", 2020-03-09T00:25:52.803 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7549000000000001e-05 2020-03-09T00:25:52.803 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.803 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.803 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.803 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831137+0000", 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.1499999999999999e-06 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831146+0000", 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.4503000000000001e-05 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.804 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831230+0000", 2020-03-09T00:25:52.805 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4256999999999999e-05 2020-03-09T00:25:52.805 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.805 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.805 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.805 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.831244+0000", 2020-03-09T00:25:52.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014134180999999999 2020-03-09T00:25:52.806 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.806 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.806 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.806 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845379+0000", 2020-03-09T00:25:52.806 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5120000000000002e-05 2020-03-09T00:25:52.807 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.807 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.807 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.807 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845424+0000", 2020-03-09T00:25:52.807 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.014725075000000001 2020-03-09T00:25:52.807 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.807 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=78, reqid=client.4751.0:302, at_version=49'13, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.943358637999999, 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621, 2020-03-09T00:25:52.808 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.809 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.809 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.809 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.809 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.809 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:52.809 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835296+0000", 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000166248 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.810 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.811 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835462+0000", 2020-03-09T00:25:52.811 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.6983000000000001e-05 2020-03-09T00:25:52.811 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.811 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.811 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.811 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835499+0000", 2020-03-09T00:25:52.811 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1426000000000001e-05 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835520+0000", 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1362000000000001e-05 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.812 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.835532+0000", 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001298925 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836831+0000", 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4944e-05 2020-03-09T00:25:52.813 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.836846+0000", 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0086294679999999995 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.814 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845475+0000", 2020-03-09T00:25:52.815 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7265000000000002e-05 2020-03-09T00:25:52.815 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.815 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.815 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.815 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845522+0000", 2020-03-09T00:25:52.815 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.010226621 2020-03-09T00:25:52.815 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/32 ECSubWrite(tid=79, reqid=client.4751.0:303, at_version=49'14, trim_to=0'0, roll_forward_to=35'10))", 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.941152950000003, 2020-03-09T00:25:52.816 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008, 2020-03-09T00:25:52.817 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.817 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.817 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.817 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.817 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.817 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:52.817 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.818 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.818 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.818 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.819 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837501+0000", 2020-03-09T00:25:52.819 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0375999999999999e-05 2020-03-09T00:25:52.819 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.819 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.820 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.820 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837562+0000", 2020-03-09T00:25:52.821 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1155e-05 2020-03-09T00:25:52.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.821 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.821 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.821 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837583+0000", 2020-03-09T00:25:52.821 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.573e-05 2020-03-09T00:25:52.821 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.822 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.822 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837599+0000", 2020-03-09T00:25:52.822 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6429999999999998e-06 2020-03-09T00:25:52.822 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.822 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.822 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.837605+0000", 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0074652130000000001 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845071+0000", 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3423e-05 2020-03-09T00:25:52.823 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.824 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.824 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845084+0000", 2020-03-09T00:25:52.824 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000468407 2020-03-09T00:25:52.824 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.824 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.824 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845552+0000", 2020-03-09T00:25:52.825 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.6518000000000002e-05 2020-03-09T00:25:52.825 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.825 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.825 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.825 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845619+0000", 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0081174650000000008 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/48 ECSubWrite(tid=80, reqid=client.4751.0:299, at_version=0'0, trim_to=0'0, roll_forward_to=49'30))", 2020-03-09T00:25:52.826 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:52.827 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 53.939286221000003, 2020-03-09T00:25:52.827 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999, 2020-03-09T00:25:52.827 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.827 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.827 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.827 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.827 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839368+0000", 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7139999999999998e-05 2020-03-09T00:25:52.828 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839425+0000", 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6959999999999999e-05 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.829 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839452+0000", 2020-03-09T00:25:52.830 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2141000000000003e-05 2020-03-09T00:25:52.830 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.830 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.830 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.830 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839484+0000", 2020-03-09T00:25:52.831 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2400000000000004e-06 2020-03-09T00:25:52.831 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.831 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839491+0000", 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.4830000000000002e-05 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839545+0000", 2020-03-09T00:25:52.832 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3210000000000001e-05 2020-03-09T00:25:52.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.833 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.833 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.839559+0000", 2020-03-09T00:25:52.833 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0054244760000000001 2020-03-09T00:25:52.833 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.833 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.834 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.834 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.844983+0000", 2020-03-09T00:25:52.834 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.9245000000000005e-05 2020-03-09T00:25:52.834 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.834 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.835 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.835 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:24:58.845052+0000", 2020-03-09T00:25:52.835 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0056842419999999999 2020-03-09T00:25:52.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.835 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.835 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 53/52 ECSubWrite(tid=50, reqid=client.4751.0:311, at_version=53'13, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 49.990370106999997, 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444, 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.836 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.837 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.837 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.837 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.837 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:52.837 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.837 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.838 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788284+0000", 2020-03-09T00:25:52.838 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000284932 2020-03-09T00:25:52.838 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.838 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.838 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.839 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788569+0000", 2020-03-09T00:25:52.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.8404e-05 2020-03-09T00:25:52.839 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.839 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.839 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.839 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788608+0000", 2020-03-09T00:25:52.839 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3697e-05 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788631+0000", 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1178e-05 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.840 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788642+0000", 2020-03-09T00:25:52.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000149171 2020-03-09T00:25:52.841 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.841 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.841 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.841 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788792+0000", 2020-03-09T00:25:52.841 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7829000000000002e-05 2020-03-09T00:25:52.842 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.842 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.842 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.788809+0000", 2020-03-09T00:25:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0065850479999999996 2020-03-09T00:25:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795395+0000", 2020-03-09T00:25:52.844 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5185000000000001e-05 2020-03-09T00:25:52.845 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.845 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.845 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.845 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.795450+0000", 2020-03-09T00:25:52.845 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007165444 2020-03-09T00:25:52.846 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.846 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.846 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.846 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:52.846 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:52.847 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 49.985710388000001, 2020-03-09T00:25:52.847 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999, 2020-03-09T00:25:52.847 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.847 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.848 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.848 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.848 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:52.848 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.792944+0000", 2020-03-09T00:25:52.849 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000391498 2020-03-09T00:25:52.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.849 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.849 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.849 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793335+0000", 2020-03-09T00:25:52.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.2011999999999998e-05 2020-03-09T00:25:52.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.850 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.850 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793367+0000", 2020-03-09T00:25:52.850 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4936000000000001e-05 2020-03-09T00:25:52.850 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.851 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793382+0000", 2020-03-09T00:25:52.851 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0244000000000001e-05 2020-03-09T00:25:52.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.851 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.851 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793393+0000", 2020-03-09T00:25:52.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011101499999999999 2020-03-09T00:25:52.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.852 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.852 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.852 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793504+0000", 2020-03-09T00:25:52.852 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5529e-05 2020-03-09T00:25:52.853 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.853 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.853 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.853 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.793519+0000", 2020-03-09T00:25:52.853 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057666640000000003 2020-03-09T00:25:52.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.854 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.854 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799286+0000", 2020-03-09T00:25:52.854 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011186100000000001 2020-03-09T00:25:52.854 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.854 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.855 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.855 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.799398+0000", 2020-03-09T00:25:52.855 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0064537589999999999 2020-03-09T00:25:52.855 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.855 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.855 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.855 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 49.927872031, 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001, 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.856 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.857 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:52.857 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.857 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.857 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.857 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.858 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.850782+0000", 2020-03-09T00:25:52.858 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.001481797 2020-03-09T00:25:52.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.859 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.859 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852264+0000", 2020-03-09T00:25:52.859 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.3690999999999999e-05 2020-03-09T00:25:52.859 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.859 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.860 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852308+0000", 2020-03-09T00:25:52.860 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.057e-05 2020-03-09T00:25:52.860 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.860 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.860 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.861 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852338+0000", 2020-03-09T00:25:52.861 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0482999999999999e-05 2020-03-09T00:25:52.861 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.861 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.861 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.862 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852349+0000", 2020-03-09T00:25:52.862 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.5451999999999994e-05 2020-03-09T00:25:52.862 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.862 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.863 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852434+0000", 2020-03-09T00:25:52.863 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5016e-05 2020-03-09T00:25:52.863 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.863 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.863 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.863 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.852449+0000", 2020-03-09T00:25:52.864 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.022365531000000001 2020-03-09T00:25:52.864 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.864 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874815+0000", 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7504999999999999e-05 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:02.874872+0000", 2020-03-09T00:25:52.865 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.024090045000000001 2020-03-09T00:25:52.866 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.866 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.866 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.866 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.866 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.866 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=93, reqid=client.4751.0:380, at_version=54'42, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:52.866 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 48.960616801999997, 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001, 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.867 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:52.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.868 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.868 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.868 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818038+0000", 2020-03-09T00:25:52.868 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.000153535 2020-03-09T00:25:52.868 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.869 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818191+0000", 2020-03-09T00:25:52.869 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6421e-05 2020-03-09T00:25:52.869 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.869 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.869 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.870 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818218+0000", 2020-03-09T00:25:52.870 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7093000000000001e-05 2020-03-09T00:25:52.870 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.870 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.870 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818235+0000", 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.7960000000000001e-06 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818243+0000", 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00011362 2020-03-09T00:25:52.871 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818357+0000", 2020-03-09T00:25:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2011000000000001e-05 2020-03-09T00:25:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.872 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.818369+0000", 2020-03-09T00:25:52.873 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0092246010000000007 2020-03-09T00:25:52.873 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.873 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.873 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.873 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827594+0000", 2020-03-09T00:25:52.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1653000000000002e-05 2020-03-09T00:25:52.874 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.874 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.874 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.874 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827645+0000", 2020-03-09T00:25:52.874 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0096077300000000001 2020-03-09T00:25:52.874 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.875 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.875 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.875 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.875 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.875 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/48 ECSubWrite(tid=94, reqid=client.4751.0:381, at_version=54'43, trim_to=0'0, roll_forward_to=53'40))", 2020-03-09T00:25:52.875 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:52.875 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 48.959322043999997, 2020-03-09T00:25:52.876 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005, 2020-03-09T00:25:52.876 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.876 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.876 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.876 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.876 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:52.877 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.877 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.877 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.877 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.877 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819332+0000", 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00018645000000000001 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819519+0000", 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.3156e-05 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.878 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819542+0000", 2020-03-09T00:25:52.879 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5676e-05 2020-03-09T00:25:52.879 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.879 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.879 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.879 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819558+0000", 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.742e-06 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.819564+0000", 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0027410770000000002 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.880 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822305+0000", 2020-03-09T00:25:52.881 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5634999999999999e-05 2020-03-09T00:25:52.881 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.881 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.881 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.881 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.822321+0000", 2020-03-09T00:25:52.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0053474569999999999 2020-03-09T00:25:52.882 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.882 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.882 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.882 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827669+0000", 2020-03-09T00:25:52.882 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.4604999999999998e-05 2020-03-09T00:25:52.883 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.883 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.884 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.884 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:03.827703+0000", 2020-03-09T00:25:52.884 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0083707980000000005 2020-03-09T00:25:52.884 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.884 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.885 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.885 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.885 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.886 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 54/32 ECSubWrite(tid=105, reqid=client.4751.0:427, at_version=54'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:52.886 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:52.886 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 48.555903489999999, 2020-03-09T00:25:52.886 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008, 2020-03-09T00:25:52.887 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.887 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.887 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.887 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:52.888 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.888 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.888 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.888 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.888 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222751+0000", 2020-03-09T00:25:52.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00017923099999999999 2020-03-09T00:25:52.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.889 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.889 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.889 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222930+0000", 2020-03-09T00:25:52.889 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.4340000000000001e-05 2020-03-09T00:25:52.889 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222954+0000", 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4165000000000001e-05 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.890 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222969+0000", 2020-03-09T00:25:52.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 8.6719999999999993e-06 2020-03-09T00:25:52.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.891 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.891 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.891 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.222977+0000", 2020-03-09T00:25:52.891 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7257999999999998e-05 2020-03-09T00:25:52.891 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223055+0000", 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4771e-05 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.892 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.223069+0000", 2020-03-09T00:25:52.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.008721477 2020-03-09T00:25:52.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.893 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.893 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.893 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231791+0000", 2020-03-09T00:25:52.893 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3718000000000003e-05 2020-03-09T00:25:52.893 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:04.231845+0000", 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0090936320000000008 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.894 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 63/32 ECSubWrite(tid=162, reqid=osd.11.0:162, at_version=63'46, trim_to=0'0, roll_forward_to=60'44))", 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 40.002028983000002, 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001, 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.895 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.896 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:52.896 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.896 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.896 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.896 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776625+0000", 2020-03-09T00:25:52.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.5686999999999997e-05 2020-03-09T00:25:52.897 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.897 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.897 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.897 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776681+0000", 2020-03-09T00:25:52.897 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2730000000000001e-05 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776704+0000", 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4839e-05 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.898 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776719+0000", 2020-03-09T00:25:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 7.7049999999999999e-06 2020-03-09T00:25:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.776726+0000", 2020-03-09T00:25:52.899 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00107576 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777802+0000", 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2907e-05 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.900 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777815+0000", 2020-03-09T00:25:52.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0051422669999999998 2020-03-09T00:25:52.901 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.901 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.901 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.901 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782957+0000", 2020-03-09T00:25:52.901 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.1214e-05 2020-03-09T00:25:52.902 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.902 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.902 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.902 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.783019+0000", 2020-03-09T00:25:52.902 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063931090000000001 2020-03-09T00:25:52.902 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.903 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.903 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.903 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.903 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.903 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.es4 63/32 ECSubWrite(tid=110, reqid=client.4751.0:522, at_version=63'73, trim_to=0'0, roll_forward_to=60'71))", 2020-03-09T00:25:52.903 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:52.903 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 40.001540998999999, 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001, 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:52.904 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.905 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.905 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777113+0000", 2020-03-09T00:25:52.905 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.7158999999999999e-05 2020-03-09T00:25:52.905 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.905 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777171+0000", 2020-03-09T00:25:52.906 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.0669000000000001e-05 2020-03-09T00:25:52.906 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.906 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.906 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.906 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777191+0000", 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5477000000000001e-05 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777207+0000", 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2890000000000003e-06 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.907 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777213+0000", 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00010217600000000001 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777315+0000", 2020-03-09T00:25:52.908 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2303999999999999e-05 2020-03-09T00:25:52.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.909 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.909 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.777327+0000", 2020-03-09T00:25:52.909 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.005510989 2020-03-09T00:25:52.909 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.909 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782838+0000", 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.7664000000000003e-05 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.782886+0000", 2020-03-09T00:25:52.910 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0057727270000000001 2020-03-09T00:25:52.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.911 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.911 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.911 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.911 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.911 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.3s2 63/54 ECSubWrite(tid=30, reqid=osd.0.0:30, at_version=63'40, trim_to=0'0, roll_forward_to=60'38))", 2020-03-09T00:25:52.911 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 40.000093761999999, 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004, 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.912 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:52.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.913 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.913 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.913 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778561+0000", 2020-03-09T00:25:52.913 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.6827999999999999e-05 2020-03-09T00:25:52.913 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778617+0000", 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.1481999999999999e-05 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.914 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778639+0000", 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.4163e-05 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778653+0000", 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.3899999999999998e-06 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.915 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778659+0000", 2020-03-09T00:25:52.916 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.7515000000000004e-05 2020-03-09T00:25:52.916 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.916 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.916 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.916 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778757+0000", 2020-03-09T00:25:52.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2418e-05 2020-03-09T00:25:52.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.917 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.917 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.917 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.778769+0000", 2020-03-09T00:25:52.917 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0058080179999999999 2020-03-09T00:25:52.917 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.918 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784577+0000", 2020-03-09T00:25:52.918 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.3770000000000002e-05 2020-03-09T00:25:52.918 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.918 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.918 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.919 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.784631+0000", 2020-03-09T00:25:52.919 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060705840000000004 2020-03-09T00:25:52.919 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.919 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.919 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.919 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.919 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 63/54 ECSubWrite(tid=85, reqid=osd.14.0:85, at_version=63'28, trim_to=0'0, roll_forward_to=62'26))", 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 39.998769959000001, 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001, 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.920 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.921 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.921 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:52.921 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.921 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.922 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.922 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779884+0000", 2020-03-09T00:25:52.922 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.9020000000000001e-05 2020-03-09T00:25:52.922 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.922 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779943+0000", 2020-03-09T00:25:52.923 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.8972e-05 2020-03-09T00:25:52.923 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.923 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.923 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.923 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779972+0000", 2020-03-09T00:25:52.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.5282999999999999e-05 2020-03-09T00:25:52.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.924 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.924 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.924 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779988+0000", 2020-03-09T00:25:52.924 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.5200000000000003e-06 2020-03-09T00:25:52.924 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.779994+0000", 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00027147400000000001 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.925 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780266+0000", 2020-03-09T00:25:52.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.1885000000000001e-05 2020-03-09T00:25:52.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.926 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.926 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.926 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.780278+0000", 2020-03-09T00:25:52.926 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0050865659999999998 2020-03-09T00:25:52.926 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785364+0000", 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.6227999999999999e-05 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.927 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:12.785410+0000", 2020-03-09T00:25:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0055259480000000001 2020-03-09T00:25:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.928 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 37.976301014999997, 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020881999999999, 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.929 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.802353+0000", 2020-03-09T00:25:52.930 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0028740570000000002 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805227+0000", 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.9300999999999999e-05 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.931 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805257+0000", 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2918000000000001e-05 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805270+0000", 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.5249999999999999e-06 2020-03-09T00:25:52.932 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805274+0000", 2020-03-09T00:25:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8279000000000002e-05 2020-03-09T00:25:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805332+0000", 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3175e-05 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.805346+0000", 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.011977272000000001 2020-03-09T00:25:52.934 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.935 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.935 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.937 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.817323+0000", 2020-03-09T00:25:52.937 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.1354999999999999e-05 2020-03-09T00:25:52.937 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:14.817374+0000", 2020-03-09T00:25:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.015020881999999999 2020-03-09T00:25:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.938 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.7s4 68/67 ECSubWrite(tid=96, reqid=client.4751.0:565, at_version=68'38, trim_to=0'0, roll_forward_to=68'36))", 2020-03-09T00:25:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 34.828495504000003, 2020-03-09T00:25:52.939 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063074259999999997, 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.940 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.941 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950159+0000", 2020-03-09T00:25:52.941 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 9.2433000000000001e-05 2020-03-09T00:25:52.941 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.941 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.941 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950251+0000", 2020-03-09T00:25:52.942 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.2212e-05 2020-03-09T00:25:52.942 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.942 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.942 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.942 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950274+0000", 2020-03-09T00:25:52.943 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.666e-05 2020-03-09T00:25:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950290+0000", 2020-03-09T00:25:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2519999999999996e-06 2020-03-09T00:25:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.944 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950296+0000", 2020-03-09T00:25:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8819000000000001e-05 2020-03-09T00:25:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.945 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.946 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950345+0000", 2020-03-09T00:25:52.946 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.2407e-05 2020-03-09T00:25:52.946 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.946 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.946 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.946 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.950358+0000", 2020-03-09T00:25:52.946 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0060565740000000003 2020-03-09T00:25:52.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.947 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.947 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.947 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956414+0000", 2020-03-09T00:25:52.947 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.2068999999999999e-05 2020-03-09T00:25:52.947 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.948 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.948 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.948 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.956466+0000", 2020-03-09T00:25:52.948 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0063074259999999997 2020-03-09T00:25:52.948 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.948 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.948 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.949 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.949 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.950 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 68/67 ECSubWrite(tid=203, reqid=client.4751.0:575, at_version=68'66, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:52.950 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:52.950 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 34.825280192999998, 2020-03-09T00:25:52.951 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007455082, 2020-03-09T00:25:52.951 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.951 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.951 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.951 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.951 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.951 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:52.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.952 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.952 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.952 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953374+0000", 2020-03-09T00:25:52.952 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.00038396300000000002 2020-03-09T00:25:52.952 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.953 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953758+0000", 2020-03-09T00:25:52.953 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 2.6554e-05 2020-03-09T00:25:52.953 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.953 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.953 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.954 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953785+0000", 2020-03-09T00:25:52.954 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.7886000000000001e-05 2020-03-09T00:25:52.954 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.954 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.954 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.955 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953803+0000", 2020-03-09T00:25:52.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.2519999999999996e-06 2020-03-09T00:25:52.955 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.955 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.955 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.955 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953809+0000", 2020-03-09T00:25:52.955 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.9489e-05 2020-03-09T00:25:52.956 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.957 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.957 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.957 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953858+0000", 2020-03-09T00:25:52.957 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.3121e-05 2020-03-09T00:25:52.957 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.958 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.958 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.958 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.953871+0000", 2020-03-09T00:25:52.958 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0068970350000000001 2020-03-09T00:25:52.958 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.959 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.959 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960768+0000", 2020-03-09T00:25:52.959 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 6.0782000000000002e-05 2020-03-09T00:25:52.959 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.959 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.959 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.960 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960829+0000", 2020-03-09T00:25:52.960 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.007455082 2020-03-09T00:25:52.960 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.960 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.960 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.960 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.960 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.961 INFO:teuthology.orchestra.run.smithi139.stdout: "description": "MOSDECSubOpWrite(3.cs5 68/67 ECSubWrite(tid=205, reqid=client.4751.0:577, at_version=68'68, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:52.961 INFO:teuthology.orchestra.run.smithi139.stdout: "initiated_at": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:52.961 INFO:teuthology.orchestra.run.smithi139.stdout: "age": 34.817928164999998, 2020-03-09T00:25:52.961 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0061821239999999998, 2020-03-09T00:25:52.961 INFO:teuthology.orchestra.run.smithi139.stdout: "type_data": { 2020-03-09T00:25:52.961 INFO:teuthology.orchestra.run.smithi139.stdout: "flag_point": "reached pg", 2020-03-09T00:25:52.961 INFO:teuthology.orchestra.run.smithi139.stdout: "events": [ 2020-03-09T00:25:52.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "initiated", 2020-03-09T00:25:52.962 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:52.962 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0 2020-03-09T00:25:52.962 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.962 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.962 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "throttled", 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.960726+0000", 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0023476109999999999 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "header_read", 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963074+0000", 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.8515000000000001e-05 2020-03-09T00:25:52.963 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.964 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "all_read", 2020-03-09T00:25:52.964 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963092+0000", 2020-03-09T00:25:52.964 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.0417999999999999e-05 2020-03-09T00:25:52.964 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.964 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.964 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "dispatched", 2020-03-09T00:25:52.965 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963103+0000", 2020-03-09T00:25:52.965 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 5.8919999999999997e-06 2020-03-09T00:25:52.965 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.965 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.965 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "queued_for_pg", 2020-03-09T00:25:52.965 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963109+0000", 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 3.9573000000000002e-05 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "reached_pg", 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963148+0000", 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 1.181e-05 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.966 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_started", 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.963160+0000", 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0037000549999999998 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "sub_op_committed", 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.966860+0000", 2020-03-09T00:25:52.967 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 4.8250000000000001e-05 2020-03-09T00:25:52.968 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2020-03-09T00:25:52.968 INFO:teuthology.orchestra.run.smithi139.stdout: { 2020-03-09T00:25:52.968 INFO:teuthology.orchestra.run.smithi139.stdout: "event": "done", 2020-03-09T00:25:52.968 INFO:teuthology.orchestra.run.smithi139.stdout: "time": "2020-03-09T00:25:17.966908+0000", 2020-03-09T00:25:52.968 INFO:teuthology.orchestra.run.smithi139.stdout: "duration": 0.0061821239999999998 2020-03-09T00:25:52.968 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.968 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.969 INFO:teuthology.orchestra.run.smithi139.stdout: } 2020-03-09T00:25:52.969 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2020-03-09T00:25:52.969 INFO:teuthology.orchestra.run.smithi139.stdout:} 2020-03-09T00:25:52.971 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-03-09T00:25:53.108 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:53.108 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:53.109 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:53.109 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:53.118 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-03-09T00:25:53.241 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:53.242 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:53.242 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:53.242 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:53.242 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:53.251 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-03-09T00:25:53.380 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:200 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~557056] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:53.381 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.543445241000001, 2020-03-09T00:25:53.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845, 2020-03-09T00:25:53.382 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.382 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.382 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.382 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.382 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.382 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 200 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.834723+0000", 2020-03-09T00:25:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001017256 2020-03-09T00:25:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835741+0000", 2020-03-09T00:25:53.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0451999999999999e-05 2020-03-09T00:25:53.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835751+0000", 2020-03-09T00:25:53.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1429000000000001e-05 2020-03-09T00:25:53.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835773+0000", 2020-03-09T00:25:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0686000000000001e-05 2020-03-09T00:25:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835783+0000", 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7098e-05 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835850+0000", 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100177 2020-03-09T00:25:53.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.835951+0000", 2020-03-09T00:25:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014653610000000001 2020-03-09T00:25:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837416+0000", 2020-03-09T00:25:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082709510000000003 2020-03-09T00:25:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845687+0000", 2020-03-09T00:25:53.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015616149999999999 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847248+0000", 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6819999999999998e-05 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847345+0000", 2020-03-09T00:25:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012621845 2020-03-09T00:25:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:201 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.540736937999995, 2020-03-09T00:25:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784, 2020-03-09T00:25:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.393 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 201 2020-03-09T00:25:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.837432+0000", 2020-03-09T00:25:53.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081556599999999999 2020-03-09T00:25:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838247+0000", 2020-03-09T00:25:53.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3170000000000004e-06 2020-03-09T00:25:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838255+0000", 2020-03-09T00:25:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0073e-05 2020-03-09T00:25:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838265+0000", 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2199999999999997e-06 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838271+0000", 2020-03-09T00:25:53.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003565814 2020-03-09T00:25:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841837+0000", 2020-03-09T00:25:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013320899999999999 2020-03-09T00:25:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841970+0000", 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001092275 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.843062+0000", 2020-03-09T00:25:53.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003950617 2020-03-09T00:25:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847013+0000", 2020-03-09T00:25:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067726199999999998 2020-03-09T00:25:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847690+0000", 2020-03-09T00:25:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000361431 2020-03-09T00:25:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.848052+0000", 2020-03-09T00:25:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010619784 2020-03-09T00:25:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.405 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:202 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc 4=[4] ondisk+write+known_if_redirected e38)", 2020-03-09T00:25:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.539850018999999, 2020-03-09T00:25:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999, 2020-03-09T00:25:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.406 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 202 2020-03-09T00:25:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838319+0000", 2020-03-09T00:25:53.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3191e-05 2020-03-09T00:25:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838362+0000", 2020-03-09T00:25:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2454e-05 2020-03-09T00:25:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838374+0000", 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8269999999999998e-06 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838382+0000", 2020-03-09T00:25:53.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1470000000000003e-06 2020-03-09T00:25:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838386+0000", 2020-03-09T00:25:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073633309999999999 2020-03-09T00:25:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845750+0000", 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103528 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845853+0000", 2020-03-09T00:25:53.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059542799999999995 2020-03-09T00:25:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846449+0000", 2020-03-09T00:25:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056665000000000003 2020-03-09T00:25:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847015+0000", 2020-03-09T00:25:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022036690000000001 2020-03-09T00:25:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849219+0000", 2020-03-09T00:25:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7259e-05 2020-03-09T00:25:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849296+0000", 2020-03-09T00:25:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010977483999999999 2020-03-09T00:25:53.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:203 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e38)", 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.539755040000003, 2020-03-09T00:25:53.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296, 2020-03-09T00:25:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 203 2020-03-09T00:25:53.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838414+0000", 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4102000000000001e-05 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838438+0000", 2020-03-09T00:25:53.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4809999999999998e-06 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838441+0000", 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.691e-06 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838448+0000", 2020-03-09T00:25:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9409999999999999e-06 2020-03-09T00:25:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.838451+0000", 2020-03-09T00:25:53.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084379850000000003 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.846889+0000", 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119473 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.847008+0000", 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029378160000000002 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.849946+0000", 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129367 2020-03-09T00:25:53.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.850076+0000", 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013511040000000001 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.851427+0000", 2020-03-09T00:25:53.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01301296 2020-03-09T00:25:53.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.425 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.425 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.536458581999995, 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999, 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841710+0000", 2020-03-09T00:25:53.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8059000000000002e-05 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841758+0000", 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3980000000000005e-06 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841768+0000", 2020-03-09T00:25:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4708e-05 2020-03-09T00:25:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841782+0000", 2020-03-09T00:25:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1330000000000001e-06 2020-03-09T00:25:53.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841789+0000", 2020-03-09T00:25:53.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037343680000000001 2020-03-09T00:25:53.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845524+0000", 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9321e-05 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845543+0000", 2020-03-09T00:25:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038329869999999999 2020-03-09T00:25:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.536302441999993, 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002, 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841866+0000", 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4193e-05 2020-03-09T00:25:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841930+0000", 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0671e-05 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841941+0000", 2020-03-09T00:25:53.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5038000000000001e-05 2020-03-09T00:25:53.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841956+0000", 2020-03-09T00:25:53.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1e-06 2020-03-09T00:25:53.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.841962+0000", 2020-03-09T00:25:53.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036100849999999999 2020-03-09T00:25:53.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845572+0000", 2020-03-09T00:25:53.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117707 2020-03-09T00:25:53.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845690+0000", 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038237940000000002 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=15, last_complete=38'8, committed=1, applied=1))", 2020-03-09T00:25:53.439 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:53.440 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 65.535796762000004, 2020-03-09T00:25:53.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002, 2020-03-09T00:25:53.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.440 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.440 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842372+0000", 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6950000000000002e-05 2020-03-09T00:25:53.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842429+0000", 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1120000000000002e-06 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842438+0000", 2020-03-09T00:25:53.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5486999999999999e-05 2020-03-09T00:25:53.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842453+0000", 2020-03-09T00:25:53.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5080000000000002e-06 2020-03-09T00:25:53.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.842460+0000", 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003166791 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845627+0000", 2020-03-09T00:25:53.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7489000000000001e-05 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:47.845644+0000", 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032723370000000002 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.446 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(osd.12.24:1 3.8s0 3:15387ad6:::smithi10428818-43:head [assert-version v5,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-03-09T00:25:53.446 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:53.446 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.526061716000001, 2020-03-09T00:25:53.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004, 2020-03-09T00:25:53.446 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852107+0000", 2020-03-09T00:25:53.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0395000000000006e-05 2020-03-09T00:25:53.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852177+0000", 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1959999999999996e-06 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852184+0000", 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0183000000000001e-05 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852194+0000", 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.463e-06 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852199+0000", 2020-03-09T00:25:53.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7527e-05 2020-03-09T00:25:53.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852257+0000", 2020-03-09T00:25:53.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132466 2020-03-09T00:25:53.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.852389+0000", 2020-03-09T00:25:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042799869999999999 2020-03-09T00:25:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.856669+0000", 2020-03-09T00:25:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045622170000000004 2020-03-09T00:25:53.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.454 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.454 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=56, reqid=client.4751.0:251, at_version=44'11, trim_to=0'0, roll_forward_to=35'9))", 2020-03-09T00:25:53.454 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.515131629999999, 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004, 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863037+0000", 2020-03-09T00:25:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022289400000000001 2020-03-09T00:25:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863260+0000", 2020-03-09T00:25:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0071000000000001e-05 2020-03-09T00:25:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863280+0000", 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2445e-05 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863292+0000", 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2320000000000001e-06 2020-03-09T00:25:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863298+0000", 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7059999999999998e-05 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863365+0000", 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6728e-05 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.863381+0000", 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048764250000000002 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868258+0000", 2020-03-09T00:25:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0857999999999999e-05 2020-03-09T00:25:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:53.868319+0000", 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052817130000000004 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 48/32 ECSubWrite(tid=43, reqid=osd.14.0:43, at_version=48'12, trim_to=0'0, roll_forward_to=44'10))", 2020-03-09T00:25:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.605554329, 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001, 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772614+0000", 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0922999999999999e-05 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772675+0000", 2020-03-09T00:25:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8955999999999999e-05 2020-03-09T00:25:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772704+0000", 2020-03-09T00:25:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6671e-05 2020-03-09T00:25:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772721+0000", 2020-03-09T00:25:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5750999999999999e-05 2020-03-09T00:25:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772737+0000", 2020-03-09T00:25:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0823999999999998e-05 2020-03-09T00:25:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772807+0000", 2020-03-09T00:25:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5262000000000001e-05 2020-03-09T00:25:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.772823+0000", 2020-03-09T00:25:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047643729999999997 2020-03-09T00:25:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777587+0000", 2020-03-09T00:25:53.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8967999999999993e-05 2020-03-09T00:25:53.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.777666+0000", 2020-03-09T00:25:53.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050517280000000001 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:275 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~704512] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:53.472 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.593262596000002, 2020-03-09T00:25:53.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002, 2020-03-09T00:25:53.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.473 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.473 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.473 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 275 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.784906+0000", 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013228269999999999 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786229+0000", 2020-03-09T00:25:53.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9299999999999997e-06 2020-03-09T00:25:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786236+0000", 2020-03-09T00:25:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3371000000000003e-05 2020-03-09T00:25:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786269+0000", 2020-03-09T00:25:53.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0840000000000003e-06 2020-03-09T00:25:53.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786276+0000", 2020-03-09T00:25:53.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7576e-05 2020-03-09T00:25:53.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786324+0000", 2020-03-09T00:25:53.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010013699999999999 2020-03-09T00:25:53.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.786424+0000", 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00139653 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787821+0000", 2020-03-09T00:25:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089582080000000005 2020-03-09T00:25:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796779+0000", 2020-03-09T00:25:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036565499999999999 2020-03-09T00:25:53.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797144+0000", 2020-03-09T00:25:53.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116684 2020-03-09T00:25:53.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797261+0000", 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012355002 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.483 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:276 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~376832] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.590924979, 2020-03-09T00:25:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001, 2020-03-09T00:25:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.485 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.486 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 276 2020-03-09T00:25:53.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.486 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:53.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787244+0000", 2020-03-09T00:25:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000487605 2020-03-09T00:25:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787731+0000", 2020-03-09T00:25:53.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.875e-06 2020-03-09T00:25:53.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787739+0000", 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8544000000000002e-05 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787778+0000", 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4640999999999999e-05 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787792+0000", 2020-03-09T00:25:53.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038656099999999998 2020-03-09T00:25:53.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791658+0000", 2020-03-09T00:25:53.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7368999999999996e-05 2020-03-09T00:25:53.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.791745+0000", 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093415700000000002 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792680+0000", 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004100289 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796780+0000", 2020-03-09T00:25:53.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074439000000000005 2020-03-09T00:25:53.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797524+0000", 2020-03-09T00:25:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2327999999999998e-05 2020-03-09T00:25:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797577+0000", 2020-03-09T00:25:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010332808000000001 2020-03-09T00:25:53.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:277 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (56) in=7b] snapc c=[c,5,4] ondisk+write+known_if_redirected e48)", 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.590281498000003, 2020-03-09T00:25:53.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006, 2020-03-09T00:25:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 277 2020-03-09T00:25:53.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.499 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:53.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787887+0000", 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010529100000000001 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787992+0000", 2020-03-09T00:25:53.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.826e-06 2020-03-09T00:25:53.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.787997+0000", 2020-03-09T00:25:53.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0430000000000008e-06 2020-03-09T00:25:53.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788005+0000", 2020-03-09T00:25:53.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1422e-05 2020-03-09T00:25:53.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788017+0000", 2020-03-09T00:25:53.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073510989999999998 2020-03-09T00:25:53.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795368+0000", 2020-03-09T00:25:53.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000159238 2020-03-09T00:25:53.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.795527+0000", 2020-03-09T00:25:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073327099999999997 2020-03-09T00:25:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796260+0000", 2020-03-09T00:25:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052061200000000005 2020-03-09T00:25:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796781+0000", 2020-03-09T00:25:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095208999999999995 2020-03-09T00:25:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797733+0000", 2020-03-09T00:25:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0259000000000001e-05 2020-03-09T00:25:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.797783+0000", 2020-03-09T00:25:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098961510000000006 2020-03-09T00:25:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:278 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-03-09T00:25:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.590088741000002, 2020-03-09T00:25:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999, 2020-03-09T00:25:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.510 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:53.510 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.510 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.510 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.510 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 278 2020-03-09T00:25:53.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.510 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788080+0000", 2020-03-09T00:25:53.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3933e-05 2020-03-09T00:25:53.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788134+0000", 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2150000000000007e-06 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788143+0000", 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.507e-05 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788158+0000", 2020-03-09T00:25:53.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.772e-05 2020-03-09T00:25:53.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.788176+0000", 2020-03-09T00:25:53.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010090758 2020-03-09T00:25:53.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798267+0000", 2020-03-09T00:25:53.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012688700000000001 2020-03-09T00:25:53.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.798394+0000", 2020-03-09T00:25:53.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001284739 2020-03-09T00:25:53.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.799678+0000", 2020-03-09T00:25:53.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011598321999999999 2020-03-09T00:25:53.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.517 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.519 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:25:53.519 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:53.519 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.586115563, 2020-03-09T00:25:53.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004, 2020-03-09T00:25:53.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.520 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.520 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:53.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792053+0000", 2020-03-09T00:25:53.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677000000000002e-05 2020-03-09T00:25:53.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792109+0000", 2020-03-09T00:25:53.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1342000000000001e-05 2020-03-09T00:25:53.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792120+0000", 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8834e-05 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792139+0000", 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1979e-05 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792151+0000", 2020-03-09T00:25:53.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043798459999999997 2020-03-09T00:25:53.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796531+0000", 2020-03-09T00:25:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.609e-05 2020-03-09T00:25:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796547+0000", 2020-03-09T00:25:53.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044937680000000004 2020-03-09T00:25:53.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.528 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 48/32 ECSubWriteReply(tid=22, last_complete=48'12, committed=1, applied=1))", 2020-03-09T00:25:53.528 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:53.528 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.585449259000001, 2020-03-09T00:25:53.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002, 2020-03-09T00:25:53.528 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.528 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792719+0000", 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2404999999999996e-05 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792782+0000", 2020-03-09T00:25:53.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2590000000000006e-06 2020-03-09T00:25:53.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792791+0000", 2020-03-09T00:25:53.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6633000000000002e-05 2020-03-09T00:25:53.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792808+0000", 2020-03-09T00:25:53.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6499999999999999e-06 2020-03-09T00:25:53.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.792814+0000", 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003762077 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796576+0000", 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8287e-05 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:57.796595+0000", 2020-03-09T00:25:53.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038753110000000002 2020-03-09T00:25:53.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.534 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.535 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:293 3.8s0 3:129db2f3:::smithi10428818-45:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:53.535 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:53.535 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.561581642999997, 2020-03-09T00:25:53.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052182940000000001, 2020-03-09T00:25:53.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.535 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.536 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.536 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 293 2020-03-09T00:25:53.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.536 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:53.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816587+0000", 2020-03-09T00:25:53.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5916999999999999e-05 2020-03-09T00:25:53.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816613+0000", 2020-03-09T00:25:53.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7960000000000007e-06 2020-03-09T00:25:53.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816623+0000", 2020-03-09T00:25:53.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0010000000000001e-05 2020-03-09T00:25:53.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816633+0000", 2020-03-09T00:25:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0229999999999998e-06 2020-03-09T00:25:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816639+0000", 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6136e-05 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816685+0000", 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1557000000000005e-05 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816766+0000", 2020-03-09T00:25:53.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095985199999999997 2020-03-09T00:25:53.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.817726+0000", 2020-03-09T00:25:53.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037736010000000001 2020-03-09T00:25:53.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821500+0000", 2020-03-09T00:25:53.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021410299999999999 2020-03-09T00:25:53.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821714+0000", 2020-03-09T00:25:53.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1298999999999994e-05 2020-03-09T00:25:53.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821805+0000", 2020-03-09T00:25:53.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052182940000000001 2020-03-09T00:25:53.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.545 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.546 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:294 3.8s0 3:129db2f3:::smithi10428818-45:head [rollback 5] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:53.546 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:53.546 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.561494572999997, 2020-03-09T00:25:53.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060119980000000002, 2020-03-09T00:25:53.546 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.546 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.546 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.547 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.547 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.547 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 294 2020-03-09T00:25:53.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.547 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:53.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816674+0000", 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4272000000000001e-05 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816698+0000", 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4240000000000002e-06 2020-03-09T00:25:53.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816702+0000", 2020-03-09T00:25:53.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8809999999999998e-06 2020-03-09T00:25:53.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816709+0000", 2020-03-09T00:25:53.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7520000000000001e-06 2020-03-09T00:25:53.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816711+0000", 2020-03-09T00:25:53.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019784109999999998 2020-03-09T00:25:53.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.818690+0000", 2020-03-09T00:25:53.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121984 2020-03-09T00:25:53.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.818812+0000", 2020-03-09T00:25:53.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083844200000000001 2020-03-09T00:25:53.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819650+0000", 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001850694 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821501+0000", 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010669200000000001 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822568+0000", 2020-03-09T00:25:53.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118218 2020-03-09T00:25:53.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822686+0000", 2020-03-09T00:25:53.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060119980000000002 2020-03-09T00:25:53.556 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.556 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.556 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.557 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:295 3.8s0 3:129db2f3:::smithi10428818-45:head [append 0~0] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:53.557 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:53.557 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.561432164000003, 2020-03-09T00:25:53.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067281850000000002, 2020-03-09T00:25:53.557 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:53.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:53.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.558 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 295 2020-03-09T00:25:53.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.559 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:53.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816737+0000", 2020-03-09T00:25:53.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5190000000000001e-05 2020-03-09T00:25:53.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816762+0000", 2020-03-09T00:25:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.061e-06 2020-03-09T00:25:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816765+0000", 2020-03-09T00:25:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7499999999999997e-06 2020-03-09T00:25:53.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816772+0000", 2020-03-09T00:25:53.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.588e-06 2020-03-09T00:25:53.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.816774+0000", 2020-03-09T00:25:53.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035648450000000001 2020-03-09T00:25:53.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.820339+0000", 2020-03-09T00:25:53.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113943 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.820453+0000", 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071432600000000005 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821167+0000", 2020-03-09T00:25:53.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068605100000000002 2020-03-09T00:25:53.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821853+0000", 2020-03-09T00:25:53.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001563734 2020-03-09T00:25:53.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:53.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823417+0000", 2020-03-09T00:25:53.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7697000000000001e-05 2020-03-09T00:25:53.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.823465+0000", 2020-03-09T00:25:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067281850000000002 2020-03-09T00:25:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.568 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 49/32 ECSubWriteReply(tid=27, last_complete=49'16, committed=1, applied=1))", 2020-03-09T00:25:53.569 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:53.569 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.559146394000003, 2020-03-09T00:25:53.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003135231, 2020-03-09T00:25:53.569 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:53.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.569 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:53.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:53.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:53.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:53.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:53.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819022+0000", 2020-03-09T00:25:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0222e-05 2020-03-09T00:25:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819053+0000", 2020-03-09T00:25:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8910000000000003e-06 2020-03-09T00:25:53.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:53.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819057+0000", 2020-03-09T00:25:53.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9330000000000001e-06 2020-03-09T00:25:53.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:53.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819065+0000", 2020-03-09T00:25:53.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.968e-06 2020-03-09T00:25:53.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.819069+0000", 2020-03-09T00:25:53.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023769609999999999 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.821446+0000", 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071125599999999995 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:53.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:24:58.822158+0000", 2020-03-09T00:25:53.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003135231 2020-03-09T00:25:53.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.575 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:53.575 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:53.576 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:53.577 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-03-09T00:25:53.704 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:53.705 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:53.705 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:53.705 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:53.713 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-03-09T00:25:53.828 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:53.829 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:53.829 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:53.829 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:53.829 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:53.837 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-03-09T00:25:53.912 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub 2020-03-09T00:25:53.959 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:53.959 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:53.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:53.959 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:53.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.959 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s3 53/52 ECSubWrite(tid=50, reqid=client.4751.0:311, at_version=53'13, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:53.960 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:53.960 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 51.167763074, 2020-03-09T00:25:53.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013212144, 2020-03-09T00:25:53.960 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:53.960 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.960 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:53.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:53.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:53.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:53.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.961 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.961 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:53.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788259+0000", 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00046852800000000001 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788727+0000", 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6373999999999999e-05 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:53.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788754+0000", 2020-03-09T00:25:53.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5746000000000001e-05 2020-03-09T00:25:53.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:53.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788769+0000", 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4629999999999998e-06 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788776+0000", 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00019937899999999999 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:53.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788975+0000", 2020-03-09T00:25:53.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4996e-05 2020-03-09T00:25:53.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.788990+0000", 2020-03-09T00:25:53.966 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012423663999999999 2020-03-09T00:25:53.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.801414+0000", 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6993999999999997e-05 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.801471+0000", 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013212144 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:53.968 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.968 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.968 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:53.968 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:53.968 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 51.104811040999998, 2020-03-09T00:25:53.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023044938000000001, 2020-03-09T00:25:53.969 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:53.969 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.969 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:53.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:53.970 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:53.970 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:53.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.970 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:53.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.851211+0000", 2020-03-09T00:25:53.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0059576639999999997 2020-03-09T00:25:53.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:53.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857168+0000", 2020-03-09T00:25:53.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.7962000000000001e-05 2020-03-09T00:25:53.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:53.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857196+0000", 2020-03-09T00:25:53.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6986999999999998e-05 2020-03-09T00:25:53.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:53.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857213+0000", 2020-03-09T00:25:53.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4509999999999997e-06 2020-03-09T00:25:53.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857220+0000", 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000106183 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857326+0000", 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.182e-05 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.857338+0000", 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016865054000000001 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.874203+0000", 2020-03-09T00:25:53.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.2816999999999999e-05 2020-03-09T00:25:53.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:53.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.874256+0000", 2020-03-09T00:25:53.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023044938000000001 2020-03-09T00:25:53.977 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.977 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.2s3 54/48 ECSubWrite(tid=7, reqid=client.4751.0:375, at_version=54'16, trim_to=0'0, roll_forward_to=52'14))", 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.151188339000001, 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014845443, 2020-03-09T00:25:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:53.980 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:53.980 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:53.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:53.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:53.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:53.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:53.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.804833+0000", 2020-03-09T00:25:53.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0040868809999999997 2020-03-09T00:25:53.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.982 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:53.982 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808920+0000", 2020-03-09T00:25:53.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1741e-05 2020-03-09T00:25:53.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808952+0000", 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5818e-05 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:53.983 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808968+0000", 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9209999999999998e-06 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.808977+0000", 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00018277500000000001 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:53.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.809160+0000", 2020-03-09T00:25:53.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6409999999999997e-06 2020-03-09T00:25:53.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.809169+0000", 2020-03-09T00:25:53.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010439004999999999 2020-03-09T00:25:53.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.819608+0000", 2020-03-09T00:25:53.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0661000000000007e-05 2020-03-09T00:25:53.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.819679+0000", 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014845443 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:485 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.143395882999997, 2020-03-09T00:25:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028623089000000001, 2020-03-09T00:25:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 485 2020-03-09T00:25:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812626+0000", 2020-03-09T00:25:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.0157000000000001e-05 2020-03-09T00:25:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812676+0000", 2020-03-09T00:25:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9910000000000008e-06 2020-03-09T00:25:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812685+0000", 2020-03-09T00:25:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6279000000000001e-05 2020-03-09T00:25:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812701+0000", 2020-03-09T00:25:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.7309999999999993e-06 2020-03-09T00:25:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812709+0000", 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016644500000000001 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812875+0000", 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00014119600000000001 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813017+0000", 2020-03-09T00:25:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0013810770000000001 2020-03-09T00:25:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.814398+0000", 2020-03-09T00:25:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015392095 2020-03-09T00:25:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829790+0000", 2020-03-09T00:25:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011370328000000001 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841160+0000", 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8789999999999995e-05 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841249+0000", 2020-03-09T00:25:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028623089000000001 2020-03-09T00:25:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:486 3.8s0 3:1d2a75f4:::smithi10428818-31:head [rollback 14] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.143252599999997, 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028685548000000002, 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 486 2020-03-09T00:25:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:54.001 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.001 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.001 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812769+0000", 2020-03-09T00:25:54.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4728999999999998e-05 2020-03-09T00:25:54.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.002 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.002 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812814+0000", 2020-03-09T00:25:54.002 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.1049999999999997e-06 2020-03-09T00:25:54.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.003 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812821+0000", 2020-03-09T00:25:54.003 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2251e-05 2020-03-09T00:25:54.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.003 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812833+0000", 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.754e-06 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.812840+0000", 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046243279999999996 2020-03-09T00:25:54.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817464+0000", 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121283 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.007 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.817586+0000", 2020-03-09T00:25:54.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00083277700000000004 2020-03-09T00:25:54.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.008 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.008 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.008 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.818418+0000", 2020-03-09T00:25:54.008 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011373098999999999 2020-03-09T00:25:54.008 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.009 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.009 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829791+0000", 2020-03-09T00:25:54.009 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011588105 2020-03-09T00:25:54.009 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.009 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841380+0000", 2020-03-09T00:25:54.010 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.5117000000000006e-05 2020-03-09T00:25:54.010 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.010 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.010 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.010 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.841455+0000", 2020-03-09T00:25:54.011 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028685548000000002 2020-03-09T00:25:54.011 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.011 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.011 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.011 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.012 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.012 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:487 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:54.012 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:54.012 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.142992939999999, 2020-03-09T00:25:54.012 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048834447000000003, 2020-03-09T00:25:54.013 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.013 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:54.013 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.013 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.013 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.014 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 487 2020-03-09T00:25:54.014 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.014 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.014 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.014 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.014 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:54.014 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813029+0000", 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2626999999999997e-05 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.015 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813071+0000", 2020-03-09T00:25:54.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3329999999999999e-06 2020-03-09T00:25:54.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.016 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.016 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813079+0000", 2020-03-09T00:25:54.016 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5886999999999999e-05 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813095+0000", 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.601e-06 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.017 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.018 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.813101+0000", 2020-03-09T00:25:54.018 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020620191999999999 2020-03-09T00:25:54.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.018 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833721+0000", 2020-03-09T00:25:54.019 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013959399999999999 2020-03-09T00:25:54.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.019 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.019 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.833861+0000", 2020-03-09T00:25:54.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0018544080000000001 2020-03-09T00:25:54.020 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.020 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.020 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.835715+0000", 2020-03-09T00:25:54.020 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0014671090000000001 2020-03-09T00:25:54.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.021 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.021 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.837183+0000", 2020-03-09T00:25:54.021 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.024587595 2020-03-09T00:25:54.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861770+0000", 2020-03-09T00:25:54.022 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3101000000000002e-05 2020-03-09T00:25:54.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.022 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.022 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.861863+0000", 2020-03-09T00:25:54.023 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.048834447000000003 2020-03-09T00:25:54.023 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.023 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.023 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.023 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 59/54 ECSubWriteReply(tid=5, last_complete=59'26, committed=1, applied=1))", 2020-03-09T00:25:54.024 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:54.024 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.140114570000001, 2020-03-09T00:25:54.024 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013886617, 2020-03-09T00:25:54.024 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.024 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.024 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.025 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.025 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:54.025 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.026 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.026 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815907+0000", 2020-03-09T00:25:54.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4713999999999998e-05 2020-03-09T00:25:54.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.027 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.027 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.027 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815962+0000", 2020-03-09T00:25:54.027 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6719999999999993e-06 2020-03-09T00:25:54.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815971+0000", 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5335000000000001e-05 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.028 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815986+0000", 2020-03-09T00:25:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4869999999999999e-06 2020-03-09T00:25:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.815992+0000", 2020-03-09T00:25:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013780812 2020-03-09T00:25:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.030 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829773+0000", 2020-03-09T00:25:54.030 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.0596999999999999e-05 2020-03-09T00:25:54.030 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.030 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.030 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.031 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.829794+0000", 2020-03-09T00:25:54.031 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013886617 2020-03-09T00:25:54.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.031 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.031 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.031 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.032 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.032 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 59/54 ECSubWrite(tid=89, reqid=client.4751.0:498, at_version=59'55, trim_to=0'0, roll_forward_to=59'51))", 2020-03-09T00:25:54.032 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:54.032 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 45.112543852999998, 2020-03-09T00:25:54.032 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017489998, 2020-03-09T00:25:54.032 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.032 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.033 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.843478+0000", 2020-03-09T00:25:54.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001188824 2020-03-09T00:25:54.034 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.034 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.034 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.034 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844667+0000", 2020-03-09T00:25:54.034 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5516000000000001e-05 2020-03-09T00:25:54.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.035 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.035 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844692+0000", 2020-03-09T00:25:54.035 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4897e-05 2020-03-09T00:25:54.035 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.035 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844707+0000", 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3720000000000001e-06 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.844713+0000", 2020-03-09T00:25:54.036 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012514399999999999 2020-03-09T00:25:54.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.037 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.037 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845965+0000", 2020-03-09T00:25:54.037 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3451999999999999e-05 2020-03-09T00:25:54.037 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.037 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.845978+0000", 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014935277 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860914+0000", 2020-03-09T00:25:54.038 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4219999999999999e-05 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:08.860968+0000", 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017489998 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.039 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.040 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.040 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.040 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:528 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~0] snapc 1b=[1a,15,13,c] ondisk+write+known_if_redirected e64)", 2020-03-09T00:25:54.040 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:54.040 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 40.184567878000003, 2020-03-09T00:25:54.040 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015882618000000001, 2020-03-09T00:25:54.040 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.042 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:54.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.042 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.043 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 528 2020-03-09T00:25:54.043 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.043 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.043 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.044 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.044 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:54.044 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.044 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.045 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771454+0000", 2020-03-09T00:25:54.045 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5457e-05 2020-03-09T00:25:54.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.045 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.045 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.046 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771479+0000", 2020-03-09T00:25:54.046 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8210000000000001e-06 2020-03-09T00:25:54.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.046 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.046 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771484+0000", 2020-03-09T00:25:54.047 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.2479999999999996e-06 2020-03-09T00:25:54.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.047 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.047 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.047 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771492+0000", 2020-03-09T00:25:54.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.8449999999999999e-06 2020-03-09T00:25:54.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.048 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.048 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.048 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771495+0000", 2020-03-09T00:25:54.048 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4236999999999998e-05 2020-03-09T00:25:54.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.049 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.049 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.049 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771529+0000", 2020-03-09T00:25:54.049 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5760000000000006e-06 2020-03-09T00:25:54.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.050 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.050 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:54.050 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.771538+0000", 2020-03-09T00:25:54.050 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0078539520000000008 2020-03-09T00:25:54.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.051 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.051 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.051 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.779392+0000", 2020-03-09T00:25:54.051 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011161400000000001 2020-03-09T00:25:54.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.052 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.052 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.052 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.779504+0000", 2020-03-09T00:25:54.052 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00052493799999999997 2020-03-09T00:25:54.052 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.780029+0000", 2020-03-09T00:25:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.006329693 2020-03-09T00:25:54.053 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.054 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.054 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.786358+0000", 2020-03-09T00:25:54.054 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00088499900000000003 2020-03-09T00:25:54.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.055 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.055 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.787243+0000", 2020-03-09T00:25:54.055 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.3238000000000004e-05 2020-03-09T00:25:54.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.056 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.056 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:13.787336+0000", 2020-03-09T00:25:54.056 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015882618000000001 2020-03-09T00:25:54.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.056 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.057 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.cs4 65/32 ECSubWrite(tid=191, reqid=client.4751.0:551, at_version=65'62, trim_to=0'0, roll_forward_to=65'58))", 2020-03-09T00:25:54.057 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:54.057 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 39.153612723000002, 2020-03-09T00:25:54.058 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014065318, 2020-03-09T00:25:54.058 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.058 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.058 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.058 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:54.059 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.059 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.059 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.802409+0000", 2020-03-09T00:25:54.060 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002574813 2020-03-09T00:25:54.060 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.060 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.060 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.060 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.804984+0000", 2020-03-09T00:25:54.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9797e-05 2020-03-09T00:25:54.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.061 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.061 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.061 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805004+0000", 2020-03-09T00:25:54.061 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1419999999999999e-06 2020-03-09T00:25:54.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.062 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.062 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.062 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805013+0000", 2020-03-09T00:25:54.062 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2899999999999996e-06 2020-03-09T00:25:54.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.063 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.063 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.063 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805017+0000", 2020-03-09T00:25:54.063 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2918999999999998e-05 2020-03-09T00:25:54.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.064 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.064 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805080+0000", 2020-03-09T00:25:54.064 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7710000000000001e-06 2020-03-09T00:25:54.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.064 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.065 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.805090+0000", 2020-03-09T00:25:54.065 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011347336 2020-03-09T00:25:54.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.065 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.065 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.816437+0000", 2020-03-09T00:25:54.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7249999999999997e-05 2020-03-09T00:25:54.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.066 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.066 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.066 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:14.816474+0000", 2020-03-09T00:25:54.066 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014065318 2020-03-09T00:25:54.067 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.067 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.067 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.067 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.067 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:676 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~2523136,checksum] snapc 0=[] ondisk+read+known_if_redirected e80)", 2020-03-09T00:25:54.067 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.141363946999999, 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014479270000000001, 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.068 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 676 2020-03-09T00:25:54.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.069 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.069 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.069 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.069 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:54.069 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.070 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814658+0000", 2020-03-09T00:25:54.070 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.6242000000000005e-05 2020-03-09T00:25:54.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.070 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.070 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.072 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814744+0000", 2020-03-09T00:25:54.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1100000000000003e-06 2020-03-09T00:25:54.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.073 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.073 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.073 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814752+0000", 2020-03-09T00:25:54.073 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3115999999999999e-05 2020-03-09T00:25:54.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.074 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.074 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814765+0000", 2020-03-09T00:25:54.074 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5550000000000002e-06 2020-03-09T00:25:54.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.074 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814771+0000", 2020-03-09T00:25:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0043361559999999999 2020-03-09T00:25:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.075 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.819107+0000", 2020-03-09T00:25:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013518100000000001 2020-03-09T00:25:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.819242+0000", 2020-03-09T00:25:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098949099999999998 2020-03-09T00:25:54.076 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.077 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.077 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.077 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829137+0000", 2020-03-09T00:25:54.077 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014479270000000001 2020-03-09T00:25:54.077 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.077 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.078 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.078 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.078 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.078 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:677 3.8s0 3.2fae54b8 (undecoded) ondisk+read+known_if_redirected e80)", 2020-03-09T00:25:54.078 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:54.078 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 24.141187231, 2020-03-09T00:25:54.079 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014509384, 2020-03-09T00:25:54.079 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.079 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.079 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.079 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.079 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.079 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 677 2020-03-09T00:25:54.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.080 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.080 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.080 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.080 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:54.080 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.081 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.081 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814835+0000", 2020-03-09T00:25:54.081 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.6549000000000002e-05 2020-03-09T00:25:54.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.081 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814911+0000", 2020-03-09T00:25:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.3329999999999997e-06 2020-03-09T00:25:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814919+0000", 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2802999999999999e-05 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814932+0000", 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3369999999999996e-06 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.814939+0000", 2020-03-09T00:25:54.084 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014396528 2020-03-09T00:25:54.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.084 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.084 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829335+0000", 2020-03-09T00:25:54.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.8340000000000005e-06 2020-03-09T00:25:54.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.085 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.085 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:29.829344+0000", 2020-03-09T00:25:54.085 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.014509384 2020-03-09T00:25:54.085 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:749 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~475136] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.866101+0000", 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.089920565, 2020-03-09T00:25:54.086 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01484748, 2020-03-09T00:25:54.087 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.087 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:54.087 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.087 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.087 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.087 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 749 2020-03-09T00:25:54.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.088 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.088 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.088 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866101+0000", 2020-03-09T00:25:54.088 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.089 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.089 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866101+0000", 2020-03-09T00:25:54.089 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00061777499999999999 2020-03-09T00:25:54.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.089 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866719+0000", 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1809999999999998e-06 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866725+0000", 2020-03-09T00:25:54.090 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.6569999999999992e-06 2020-03-09T00:25:54.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.091 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.091 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866735+0000", 2020-03-09T00:25:54.091 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0209999999999996e-06 2020-03-09T00:25:54.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.091 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.092 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.866741+0000", 2020-03-09T00:25:54.092 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002096213 2020-03-09T00:25:54.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.092 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.092 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.868837+0000", 2020-03-09T00:25:54.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00023417300000000001 2020-03-09T00:25:54.093 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.093 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.093 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.093 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869071+0000", 2020-03-09T00:25:54.093 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011264479999999999 2020-03-09T00:25:54.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.094 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.094 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.870198+0000", 2020-03-09T00:25:54.094 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010087149 2020-03-09T00:25:54.094 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.094 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.095 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880285+0000", 2020-03-09T00:25:54.095 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00037859299999999999 2020-03-09T00:25:54.095 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.095 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.095 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880663+0000", 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00028527000000000001 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880949+0000", 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01484748 2020-03-09T00:25:54.096 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.097 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.097 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.097 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.097 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.097 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:750 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~507904] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:54.097 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.868989+0000", 2020-03-09T00:25:54.097 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.087033043000002, 2020-03-09T00:25:54.098 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015288784, 2020-03-09T00:25:54.098 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.098 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:54.098 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.098 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.098 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.098 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 750 2020-03-09T00:25:54.099 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.099 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.099 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.099 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.868989+0000", 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.868989+0000", 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00096076899999999999 2020-03-09T00:25:54.100 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.101 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869949+0000", 2020-03-09T00:25:54.101 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0250000000000002e-06 2020-03-09T00:25:54.101 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.101 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.101 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869957+0000", 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4758e-05 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869972+0000", 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.618e-06 2020-03-09T00:25:54.102 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.103 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.869979+0000", 2020-03-09T00:25:54.103 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027946799999999999 2020-03-09T00:25:54.103 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.103 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.103 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872774+0000", 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000105646 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872879+0000", 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00082986099999999999 2020-03-09T00:25:54.104 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.105 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.873709+0000", 2020-03-09T00:25:54.105 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066888529999999998 2020-03-09T00:25:54.105 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.105 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.105 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.880398+0000", 2020-03-09T00:25:54.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0035930810000000001 2020-03-09T00:25:54.106 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.106 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.106 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.106 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883991+0000", 2020-03-09T00:25:54.106 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00028649300000000002 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884277+0000", 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015288784 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.107 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.108 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.108 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.108 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:752 3.8s0 3:1d2a75f4:::smithi10428818-31:head [append 0~98304] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:54.108 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.872378+0000", 2020-03-09T00:25:54.108 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.083643976000001, 2020-03-09T00:25:54.108 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013385007000000001, 2020-03-09T00:25:54.109 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.109 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:54.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.109 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.109 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 752 2020-03-09T00:25:54.109 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.110 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.110 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.110 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872378+0000", 2020-03-09T00:25:54.110 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.110 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.110 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872378+0000", 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015559499999999999 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872533+0000", 2020-03-09T00:25:54.111 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.8180000000000004e-06 2020-03-09T00:25:54.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.112 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.112 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872541+0000", 2020-03-09T00:25:54.112 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5211000000000001e-05 2020-03-09T00:25:54.112 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.112 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872556+0000", 2020-03-09T00:25:54.113 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5539999999999997e-06 2020-03-09T00:25:54.113 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.113 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.113 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.113 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872562+0000", 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087003029999999995 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.881262+0000", 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013599200000000001 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.881398+0000", 2020-03-09T00:25:54.115 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00081354799999999998 2020-03-09T00:25:54.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.115 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.115 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.882212+0000", 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0021470320000000001 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884359+0000", 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001083751 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.885443+0000", 2020-03-09T00:25:54.117 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00032020300000000002 2020-03-09T00:25:54.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.117 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.117 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.885763+0000", 2020-03-09T00:25:54.118 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013385007000000001 2020-03-09T00:25:54.118 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.118 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.118 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.118 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:753 3.8s0 3:1d2a75f4:::smithi10428818-31:head [setxattr _header (57) in=7b] snapc 2c=[] ondisk+write+known_if_redirected e85)", 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.872624+0000", 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.083397730000002, 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013590579, 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.119 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.120 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 753 2020-03-09T00:25:54.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.120 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.120 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.120 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.120 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872624+0000", 2020-03-09T00:25:54.120 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872624+0000", 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0097000000000002e-05 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872704+0000", 2020-03-09T00:25:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1079999999999993e-06 2020-03-09T00:25:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872712+0000", 2020-03-09T00:25:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2544000000000001e-05 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872725+0000", 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4110000000000004e-06 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.123 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872730+0000", 2020-03-09T00:25:54.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010322237999999999 2020-03-09T00:25:54.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.124 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.124 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883052+0000", 2020-03-09T00:25:54.124 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5949e-05 2020-03-09T00:25:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883138+0000", 2020-03-09T00:25:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00050229199999999999 2020-03-09T00:25:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.883641+0000", 2020-03-09T00:25:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001221462 2020-03-09T00:25:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.126 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884862+0000", 2020-03-09T00:25:54.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010869440000000001 2020-03-09T00:25:54.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.127 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.127 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.885949+0000", 2020-03-09T00:25:54.127 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000265534 2020-03-09T00:25:54.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.128 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.128 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886215+0000", 2020-03-09T00:25:54.128 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013590579 2020-03-09T00:25:54.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.128 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.129 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:754 3.8s0 3:1d2a75f4:::smithi10428818-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e85)", 2020-03-09T00:25:54.129 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.872789+0000", 2020-03-09T00:25:54.129 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.083232583000001, 2020-03-09T00:25:54.129 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016005664999999999, 2020-03-09T00:25:54.129 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 754 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.130 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872789+0000", 2020-03-09T00:25:54.131 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.131 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.131 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872789+0000", 2020-03-09T00:25:54.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6080000000000001e-05 2020-03-09T00:25:54.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.132 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.132 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872815+0000", 2020-03-09T00:25:54.132 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1710000000000001e-06 2020-03-09T00:25:54.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.133 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.133 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872820+0000", 2020-03-09T00:25:54.133 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.7790000000000001e-06 2020-03-09T00:25:54.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872829+0000", 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.4390000000000002e-06 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.872834+0000", 2020-03-09T00:25:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013904133000000001 2020-03-09T00:25:54.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.135 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.135 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886738+0000", 2020-03-09T00:25:54.135 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000145012 2020-03-09T00:25:54.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886883+0000", 2020-03-09T00:25:54.136 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0019120509999999999 2020-03-09T00:25:54.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.136 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.136 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.888795+0000", 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016005664999999999 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.1s6 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:54.137 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:34.884865+0000", 2020-03-09T00:25:54.138 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 19.071156892000001, 2020-03-09T00:25:54.138 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015800109, 2020-03-09T00:25:54.138 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.138 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.138 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.138 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.139 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884865+0000", 2020-03-09T00:25:54.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.139 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.139 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.139 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.884865+0000", 2020-03-09T00:25:54.139 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020353350000000001 2020-03-09T00:25:54.140 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.140 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.140 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.140 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886900+0000", 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.6072999999999999e-05 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886926+0000", 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6269999999999998e-05 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.141 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886943+0000", 2020-03-09T00:25:54.142 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2869999999999999e-06 2020-03-09T00:25:54.142 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.142 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.142 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.142 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886950+0000", 2020-03-09T00:25:54.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5357e-05 2020-03-09T00:25:54.143 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.143 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.143 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.143 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.886995+0000", 2020-03-09T00:25:54.143 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2044e-05 2020-03-09T00:25:54.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.144 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.144 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.887007+0000", 2020-03-09T00:25:54.144 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013517240999999999 2020-03-09T00:25:54.144 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.144 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.900524+0000", 2020-03-09T00:25:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000140502 2020-03-09T00:25:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:34.900665+0000", 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015800109 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:772 3.8s0 3:1d2a75f4:::smithi10428818-31:head [copy-from ver 24] snapc 32=[32] ondisk+write+known_if_redirected e86)", 2020-03-09T00:25:54.146 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:35.830970+0000", 2020-03-09T00:25:54.147 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 18.125051608, 2020-03-09T00:25:54.147 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037538067000000001, 2020-03-09T00:25:54.147 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.147 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:54.147 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.147 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.147 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 772 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.830970+0000", 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.148 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.149 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.149 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.830970+0000", 2020-03-09T00:25:54.149 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.1035000000000002e-05 2020-03-09T00:25:54.149 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.149 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.149 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831021+0000", 2020-03-09T00:25:54.150 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.5910000000000003e-06 2020-03-09T00:25:54.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.150 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.150 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.150 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831027+0000", 2020-03-09T00:25:54.150 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0087e-05 2020-03-09T00:25:54.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.151 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.151 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831037+0000", 2020-03-09T00:25:54.151 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1220000000000001e-06 2020-03-09T00:25:54.151 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.151 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831046+0000", 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000196751 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831243+0000", 2020-03-09T00:25:54.152 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.0415999999999996e-05 2020-03-09T00:25:54.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.153 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.153 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831333+0000", 2020-03-09T00:25:54.153 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018291963000000001 2020-03-09T00:25:54.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.849625+0000", 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018442281000000001 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.868067+0000", 2020-03-09T00:25:54.154 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000137429 2020-03-09T00:25:54.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.155 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:54.155 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.868205+0000", 2020-03-09T00:25:54.155 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00030339199999999998 2020-03-09T00:25:54.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.868508+0000", 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.037538067000000001 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.157 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:773 3.8s0 3:1d2a75f4:::smithi10428818-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e86)", 2020-03-09T00:25:54.157 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:35.831093+0000", 2020-03-09T00:25:54.157 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 18.124928734000001, 2020-03-09T00:25:54.157 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039651458000000001, 2020-03-09T00:25:54.157 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:54.157 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:54.157 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 773 2020-03-09T00:25:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831093+0000", 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831093+0000", 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5473000000000003e-05 2020-03-09T00:25:54.159 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:54.160 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831138+0000", 2020-03-09T00:25:54.160 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.5350000000000001e-06 2020-03-09T00:25:54.160 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.160 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.160 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831145+0000", 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2931e-05 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831158+0000", 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6210000000000001e-06 2020-03-09T00:25:54.161 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.162 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831164+0000", 2020-03-09T00:25:54.162 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00031384300000000001 2020-03-09T00:25:54.162 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.162 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.162 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831477+0000", 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4112e-05 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for blocked object", 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.831532+0000", 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.036296544 2020-03-09T00:25:54.163 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.164 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.164 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.867828+0000", 2020-03-09T00:25:54.164 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.9529000000000003e-05 2020-03-09T00:25:54.164 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.164 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.164 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:54.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.867918+0000", 2020-03-09T00:25:54.165 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0025560940000000001 2020-03-09T00:25:54.165 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.165 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.165 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:54.165 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.870474+0000", 2020-03-09T00:25:54.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0533999999999998e-05 2020-03-09T00:25:54.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.166 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.166 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:54.166 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.870554+0000", 2020-03-09T00:25:54.166 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000190242 2020-03-09T00:25:54.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:54.167 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:54.167 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:54.167 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:35.870744+0000", 2020-03-09T00:25:54.167 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.039651458000000001 2020-03-09T00:25:54.167 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.167 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.167 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.168 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:54.168 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:54.168 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:54.169 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-03-09T00:25:54.296 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:54.296 INFO:teuthology.orchestra.run.smithi104.stdout: "num_ops": 0 2020-03-09T00:25:54.297 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:54.305 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-03-09T00:25:54.420 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:54.421 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [], 2020-03-09T00:25:54.421 INFO:teuthology.orchestra.run.smithi104.stdout: "complaint_time": 30, 2020-03-09T00:25:54.421 INFO:teuthology.orchestra.run.smithi104.stdout: "num_blocked_ops": 0 2020-03-09T00:25:54.421 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:54.429 INFO:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-03-09T00:25:54.546 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2020-03-09T00:25:54.546 INFO:teuthology.orchestra.run.smithi104.stdout: "size": 20, 2020-03-09T00:25:54.546 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 600, 2020-03-09T00:25:54.547 INFO:teuthology.orchestra.run.smithi104.stdout: "ops": [ 2020-03-09T00:25:54.547 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.547 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 38/32 ECSubWrite(tid=10, reqid=client.4751.0:196, at_version=38'6, trim_to=0'0, roll_forward_to=35'4))", 2020-03-09T00:25:54.547 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:54.547 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 66.716103212999997, 2020-03-09T00:25:54.547 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737, 2020-03-09T00:25:54.547 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.548 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827509+0000", 2020-03-09T00:25:54.549 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000156164 2020-03-09T00:25:54.549 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.549 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.549 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.549 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827665+0000", 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.5397000000000001e-05 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827690+0000", 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6900000000000002e-05 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.550 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827727+0000", 2020-03-09T00:25:54.551 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.9529999999999999e-06 2020-03-09T00:25:54.551 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.551 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.551 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.551 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.827735+0000", 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0011533660000000001 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828889+0000", 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1542999999999999e-05 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.552 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.828900+0000", 2020-03-09T00:25:54.553 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0038471730000000002 2020-03-09T00:25:54.553 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.553 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.553 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.553 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832747+0000", 2020-03-09T00:25:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8874000000000001e-05 2020-03-09T00:25:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:24:47.832786+0000", 2020-03-09T00:25:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00527737 2020-03-09T00:25:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.3s1 53/52 ECSubWrite(tid=1, reqid=client.4751.0:330, at_version=53'15, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.750824819999998, 2020-03-09T00:25:54.555 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997, 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.556 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.557 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.557 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.792787+0000", 2020-03-09T00:25:54.557 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062438599999999999 2020-03-09T00:25:54.557 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.557 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793412+0000", 2020-03-09T00:25:54.558 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6639e-05 2020-03-09T00:25:54.558 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.558 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.558 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.558 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793438+0000", 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3116999999999999e-05 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793461+0000", 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 9.2469999999999997e-06 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.559 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793471+0000", 2020-03-09T00:25:54.560 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9004999999999994e-05 2020-03-09T00:25:54.560 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.560 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.560 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.560 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793540+0000", 2020-03-09T00:25:54.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8408000000000001e-05 2020-03-09T00:25:54.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.561 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.561 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.561 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.793558+0000", 2020-03-09T00:25:54.561 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0053623589999999997 2020-03-09T00:25:54.561 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798920+0000", 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2633000000000002e-05 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.562 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.798973+0000", 2020-03-09T00:25:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061857939999999997 2020-03-09T00:25:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=30, reqid=client.4751.0:346, at_version=53'18, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.716566757999999, 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999, 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.564 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:54.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.565 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.565 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.565 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.565 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827045+0000", 2020-03-09T00:25:54.565 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00013296799999999999 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827178+0000", 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6305e-05 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.566 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827205+0000", 2020-03-09T00:25:54.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5832999999999999e-05 2020-03-09T00:25:54.567 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.567 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.567 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.567 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827220+0000", 2020-03-09T00:25:54.567 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7170000000000008e-06 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827228+0000", 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3441999999999993e-05 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.568 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827292+0000", 2020-03-09T00:25:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4157e-05 2020-03-09T00:25:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.827306+0000", 2020-03-09T00:25:54.569 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0046893439999999998 2020-03-09T00:25:54.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.570 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.570 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.570 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.831995+0000", 2020-03-09T00:25:54.570 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3732000000000006e-05 2020-03-09T00:25:54.570 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.571 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.571 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.571 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.832059+0000", 2020-03-09T00:25:54.571 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050134979999999999 2020-03-09T00:25:54.571 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.571 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.571 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=31, reqid=client.4751.0:347, at_version=53'19, trim_to=0'0, roll_forward_to=52'16))", 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.708728618000002, 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996, 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.572 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.573 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.573 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.573 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:54.573 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.573 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.573 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.834883+0000", 2020-03-09T00:25:54.574 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00030527400000000002 2020-03-09T00:25:54.574 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.574 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.574 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.574 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835189+0000", 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4273999999999998e-05 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835213+0000", 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7130999999999999e-05 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.575 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835230+0000", 2020-03-09T00:25:54.576 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.6400000000000003e-06 2020-03-09T00:25:54.576 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.576 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.576 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.576 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835239+0000", 2020-03-09T00:25:54.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.0665000000000003e-05 2020-03-09T00:25:54.577 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.577 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.577 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.577 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835299+0000", 2020-03-09T00:25:54.577 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6062e-05 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.835315+0000", 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005161349 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.578 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840477+0000", 2020-03-09T00:25:54.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7466000000000003e-05 2020-03-09T00:25:54.579 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.579 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.579 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.579 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.840524+0000", 2020-03-09T00:25:54.579 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056408609999999996 2020-03-09T00:25:54.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.580 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.580 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.580 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.580 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.580 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/48 ECSubWrite(tid=34, reqid=client.4751.0:349, at_version=0'0, trim_to=0'0, roll_forward_to=53'21))", 2020-03-09T00:25:54.580 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:54.581 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 51.698150568000003, 2020-03-09T00:25:54.581 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001, 2020-03-09T00:25:54.581 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.581 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.581 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.581 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:54.582 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.582 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.582 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.582 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.582 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845461+0000", 2020-03-09T00:25:54.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6162000000000002e-05 2020-03-09T00:25:54.583 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.583 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.583 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.583 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845518+0000", 2020-03-09T00:25:54.583 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3681e-05 2020-03-09T00:25:54.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.584 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.584 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845531+0000", 2020-03-09T00:25:54.584 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376e-05 2020-03-09T00:25:54.584 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.584 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.585 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845547+0000", 2020-03-09T00:25:54.585 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.8040000000000002e-06 2020-03-09T00:25:54.585 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.585 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.585 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845555+0000", 2020-03-09T00:25:54.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00010135 2020-03-09T00:25:54.586 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.586 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.586 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.586 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845656+0000", 2020-03-09T00:25:54.586 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2714000000000001e-05 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.845669+0000", 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0072973140000000001 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.587 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.852966+0000", 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.2704999999999998e-05 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:02.853019+0000", 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0075571060000000001 2020-03-09T00:25:54.588 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 59/32 ECSubWrite(tid=9, reqid=client.4751.0:507, at_version=59'8, trim_to=0'0, roll_forward_to=52'6))", 2020-03-09T00:25:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:54.589 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 45.62963173, 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014134343000000001, 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:54.590 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.591 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.591 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.913980+0000", 2020-03-09T00:25:54.591 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00044913599999999999 2020-03-09T00:25:54.591 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.591 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914429+0000", 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3470000000000002e-05 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914473+0000", 2020-03-09T00:25:54.592 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4422000000000001e-05 2020-03-09T00:25:54.593 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.593 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.593 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.593 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914497+0000", 2020-03-09T00:25:54.593 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4169999999999997e-06 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914504+0000", 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.3522999999999995e-05 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.594 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914567+0000", 2020-03-09T00:25:54.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4911e-05 2020-03-09T00:25:54.595 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.595 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.595 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.595 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.914582+0000", 2020-03-09T00:25:54.595 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.013495323 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.928078+0000", 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.7141000000000002e-05 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.597 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:08.928115+0000", 2020-03-09T00:25:54.597 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.014134343000000001 2020-03-09T00:25:54.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.597 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.597 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.597 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.597 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 63/32 ECSubWrite(tid=83, reqid=osd.10.0:83, at_version=63'49, trim_to=0'0, roll_forward_to=60'47))", 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.768211151000003, 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073598240000000001, 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.598 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:54.599 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.599 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.599 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.599 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.599 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775401+0000", 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.4988000000000003e-05 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775456+0000", 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6618000000000002e-05 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.600 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775483+0000", 2020-03-09T00:25:54.601 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5108e-05 2020-03-09T00:25:54.601 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.601 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.601 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.601 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775498+0000", 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.7239999999999992e-06 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775505+0000", 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000110979 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775616+0000", 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2696e-05 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.775629+0000", 2020-03-09T00:25:54.603 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00708429 2020-03-09T00:25:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782713+0000", 2020-03-09T00:25:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7420999999999999e-05 2020-03-09T00:25:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.604 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782761+0000", 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0073598240000000001 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.605 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.606 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 63/48 ECSubWrite(tid=42, reqid=osd.15.0:42, at_version=63'29, trim_to=0'0, roll_forward_to=59'27))", 2020-03-09T00:25:54.606 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:54.606 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.767438329000001, 2020-03-09T00:25:54.606 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005556643, 2020-03-09T00:25:54.606 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.606 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.606 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.607 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776174+0000", 2020-03-09T00:25:54.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8814000000000003e-05 2020-03-09T00:25:54.608 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.608 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.608 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.608 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776233+0000", 2020-03-09T00:25:54.608 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2234000000000001e-05 2020-03-09T00:25:54.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.609 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.609 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776255+0000", 2020-03-09T00:25:54.609 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6149000000000001e-05 2020-03-09T00:25:54.609 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.609 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.610 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776271+0000", 2020-03-09T00:25:54.610 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.0389999999999999e-06 2020-03-09T00:25:54.610 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.610 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.610 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776278+0000", 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00040392100000000001 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776682+0000", 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3020000000000001e-05 2020-03-09T00:25:54.611 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776695+0000", 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0049768449999999997 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.612 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781672+0000", 2020-03-09T00:25:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8621e-05 2020-03-09T00:25:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.781730+0000", 2020-03-09T00:25:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.005556643 2020-03-09T00:25:54.613 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.614 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.614 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.614 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.614 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.614 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 63/32 ECSubWrite(tid=84, reqid=osd.10.0:84, at_version=63'51, trim_to=0'0, roll_forward_to=60'47))", 2020-03-09T00:25:54.614 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:54.615 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 41.766961772999998, 2020-03-09T00:25:54.615 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061568409999999997, 2020-03-09T00:25:54.615 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.615 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.615 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.615 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.615 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776650+0000", 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.2492999999999999e-05 2020-03-09T00:25:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776723+0000", 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2741000000000001e-05 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.617 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776746+0000", 2020-03-09T00:25:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5485000000000002e-05 2020-03-09T00:25:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776761+0000", 2020-03-09T00:25:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4679999999999997e-06 2020-03-09T00:25:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776767+0000", 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012315000000000001 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.619 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776891+0000", 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2254e-05 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.776903+0000", 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0058656510000000004 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782769+0000", 2020-03-09T00:25:54.621 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.8599000000000001e-05 2020-03-09T00:25:54.621 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.621 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.621 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.621 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:12.782807+0000", 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061568409999999997 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 68/67 ECSubWrite(tid=91, reqid=client.4751.0:580, at_version=68'59, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:54.622 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.576294040000001, 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006533311, 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.623 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.967318+0000", 2020-03-09T00:25:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00089329700000000004 2020-03-09T00:25:54.624 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968211+0000", 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.4736000000000001e-05 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.625 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968246+0000", 2020-03-09T00:25:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0851000000000001e-05 2020-03-09T00:25:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968267+0000", 2020-03-09T00:25:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.1970000000000001e-05 2020-03-09T00:25:54.626 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968279+0000", 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.6097999999999998e-05 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.627 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968325+0000", 2020-03-09T00:25:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2437e-05 2020-03-09T00:25:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.968337+0000", 2020-03-09T00:25:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054777970000000004 2020-03-09T00:25:54.628 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.973815+0000", 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.6124999999999997e-05 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.629 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.973851+0000", 2020-03-09T00:25:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006533311 2020-03-09T00:25:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.630 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 68/67 ECSubWrite(tid=101, reqid=client.4751.0:592, at_version=68'23, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.554540580999998, 2020-03-09T00:25:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01056028, 2020-03-09T00:25:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.631 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.632 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989071+0000", 2020-03-09T00:25:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00055521499999999996 2020-03-09T00:25:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989627+0000", 2020-03-09T00:25:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.4139999999999999e-05 2020-03-09T00:25:54.633 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989651+0000", 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5061e-05 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.634 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989666+0000", 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1540000000000002e-06 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989674+0000", 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.9898000000000001e-05 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.635 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989734+0000", 2020-03-09T00:25:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5279000000000001e-05 2020-03-09T00:25:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.636 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.989749+0000", 2020-03-09T00:25:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0098335769999999996 2020-03-09T00:25:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999583+0000", 2020-03-09T00:25:54.637 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8955999999999997e-05 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999632+0000", 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01056028 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 68/67 ECSubWrite(tid=102, reqid=client.4751.0:593, at_version=68'24, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 36.550598588, 2020-03-09T00:25:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066457310000000002, 2020-03-09T00:25:54.639 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.640 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993013+0000", 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00037136200000000003 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993385+0000", 2020-03-09T00:25:54.641 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2940999999999999e-05 2020-03-09T00:25:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993408+0000", 2020-03-09T00:25:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5376e-05 2020-03-09T00:25:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.642 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993423+0000", 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.6939999999999995e-06 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993431+0000", 2020-03-09T00:25:54.643 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9996e-05 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993481+0000", 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6351999999999999e-05 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.644 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.993537+0000", 2020-03-09T00:25:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0061008299999999998 2020-03-09T00:25:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999638+0000", 2020-03-09T00:25:54.645 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.1180000000000001e-05 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:17.999659+0000", 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0066457310000000002 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 69/32 ECSubWrite(tid=20, reqid=client.4751.0:599, at_version=69'19, trim_to=0'0, roll_forward_to=67'16))", 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 35.748873422999999, 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056539340000000002, 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.647 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.648 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794739+0000", 2020-03-09T00:25:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 8.1377000000000004e-05 2020-03-09T00:25:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794820+0000", 2020-03-09T00:25:54.649 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.3682000000000001e-05 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794844+0000", 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4918e-05 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.650 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794859+0000", 2020-03-09T00:25:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.4799999999999998e-06 2020-03-09T00:25:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.794865+0000", 2020-03-09T00:25:54.651 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0016106930000000001 2020-03-09T00:25:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796476+0000", 2020-03-09T00:25:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3885e-05 2020-03-09T00:25:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.652 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.796490+0000", 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.003855261 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800345+0000", 2020-03-09T00:25:54.653 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.7638e-05 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:18.800393+0000", 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0056539340000000002 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.654 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.6s5 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:29.821440+0000", 2020-03-09T00:25:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 24.722172421, 2020-03-09T00:25:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097473000000000004, 2020-03-09T00:25:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821440+0000", 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.656 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.821440+0000", 2020-03-09T00:25:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013057190000000001 2020-03-09T00:25:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.657 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822745+0000", 2020-03-09T00:25:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8926999999999999e-05 2020-03-09T00:25:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822774+0000", 2020-03-09T00:25:54.658 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6112000000000001e-05 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822790+0000", 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2632999999999999e-05 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.659 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822803+0000", 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.5909999999999997e-05 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822869+0000", 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4355999999999999e-05 2020-03-09T00:25:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.822883+0000", 2020-03-09T00:25:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0082614209999999997 2020-03-09T00:25:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.661 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.831145+0000", 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.2222000000000003e-05 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:29.831187+0000", 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0097473000000000004 2020-03-09T00:25:54.662 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpRead(3.as1 81/32 ECSubRead(tid=34, to_read={3:51058371:::smithi10428818-28:head=0,536576,0}, subchunks={3:51058371:::smithi10428818-28:head=[0,1]}, attrs_to_read=))", 2020-03-09T00:25:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:30.815219+0000", 2020-03-09T00:25:54.663 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 23.728392885000002, 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055806220000000004, 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815219+0000", 2020-03-09T00:25:54.664 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815219+0000", 2020-03-09T00:25:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6774999999999998e-05 2020-03-09T00:25:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815276+0000", 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3097e-05 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815289+0000", 2020-03-09T00:25:54.666 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7640000000000001e-05 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815307+0000", 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0458e-05 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.667 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.815317+0000", 2020-03-09T00:25:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0013303589999999999 2020-03-09T00:25:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.816648+0000", 2020-03-09T00:25:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0041522929999999996 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:30.820800+0000", 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0055806220000000004 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.669 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 94/32 ECSubWrite(tid=40, reqid=client.4751.0:909, at_version=94'38, trim_to=0'0, roll_forward_to=84'36))", 2020-03-09T00:25:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.882115+0000", 2020-03-09T00:25:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.661496569000001, 2020-03-09T00:25:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011532982000000001, 2020-03-09T00:25:54.670 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882115+0000", 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.671 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882115+0000", 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00012922000000000001 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882245+0000", 2020-03-09T00:25:54.672 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.6633000000000001e-05 2020-03-09T00:25:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882271+0000", 2020-03-09T00:25:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.7761000000000002e-05 2020-03-09T00:25:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.673 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882289+0000", 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.0010999999999999e-05 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882309+0000", 2020-03-09T00:25:54.674 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000129913 2020-03-09T00:25:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882439+0000", 2020-03-09T00:25:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4284000000000001e-05 2020-03-09T00:25:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.675 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.882453+0000", 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.01113858 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.893592+0000", 2020-03-09T00:25:54.676 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.6579999999999997e-05 2020-03-09T00:25:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.893648+0000", 2020-03-09T00:25:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.011532982000000001 2020-03-09T00:25:54.677 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 94/32 ECSubWrite(tid=42, reqid=client.4751.0:911, at_version=94'40, trim_to=0'0, roll_forward_to=84'36))", 2020-03-09T00:25:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.894407+0000", 2020-03-09T00:25:54.678 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.649205287999999, 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050594070000000001, 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894407+0000", 2020-03-09T00:25:54.679 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894407+0000", 2020-03-09T00:25:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00016815999999999999 2020-03-09T00:25:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.680 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894575+0000", 2020-03-09T00:25:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.0102000000000001e-05 2020-03-09T00:25:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894605+0000", 2020-03-09T00:25:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.8859999999999999e-05 2020-03-09T00:25:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.682 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894624+0000", 2020-03-09T00:25:54.683 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.9720000000000003e-06 2020-03-09T00:25:54.684 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894631+0000", 2020-03-09T00:25:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.000228946 2020-03-09T00:25:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.691 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894860+0000", 2020-03-09T00:25:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.4195e-05 2020-03-09T00:25:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.894874+0000", 2020-03-09T00:25:54.692 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0045424009999999997 2020-03-09T00:25:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899416+0000", 2020-03-09T00:25:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.9771000000000002e-05 2020-03-09T00:25:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.693 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899466+0000", 2020-03-09T00:25:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050594070000000001 2020-03-09T00:25:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.694 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.bs5 94/67 ECSubWrite(tid=98, reqid=client.4751.0:916, at_version=94'64, trim_to=0'0, roll_forward_to=92'62))", 2020-03-09T00:25:54.695 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.899621+0000", 2020-03-09T00:25:54.696 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.643991334000001, 2020-03-09T00:25:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050749460000000003, 2020-03-09T00:25:54.697 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.698 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899621+0000", 2020-03-09T00:25:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899621+0000", 2020-03-09T00:25:54.699 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 3.1442000000000002e-05 2020-03-09T00:25:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899652+0000", 2020-03-09T00:25:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.2291999999999999e-05 2020-03-09T00:25:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.700 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899674+0000", 2020-03-09T00:25:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.0436e-05 2020-03-09T00:25:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.701 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899685+0000", 2020-03-09T00:25:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.1989999999999999e-06 2020-03-09T00:25:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.702 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899689+0000", 2020-03-09T00:25:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.5750000000000001e-05 2020-03-09T00:25:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899735+0000", 2020-03-09T00:25:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2223e-05 2020-03-09T00:25:54.703 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.899747+0000", 2020-03-09T00:25:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0048983430000000003 2020-03-09T00:25:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.904645+0000", 2020-03-09T00:25:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.0260999999999999e-05 2020-03-09T00:25:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.904696+0000", 2020-03-09T00:25:54.705 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0050749460000000003 2020-03-09T00:25:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.as1 94/32 ECSubWrite(tid=43, reqid=client.4751.0:912, at_version=94'41, trim_to=0'0, roll_forward_to=94'39))", 2020-03-09T00:25:54.706 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.900590+0000", 2020-03-09T00:25:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.643022233, 2020-03-09T00:25:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063740280000000003, 2020-03-09T00:25:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.707 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.900590+0000", 2020-03-09T00:25:54.708 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.900590+0000", 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00019542499999999999 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.709 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.900785+0000", 2020-03-09T00:25:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.421e-05 2020-03-09T00:25:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.710 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.900809+0000", 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.6135000000000001e-05 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.900826+0000", 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.6980000000000001e-06 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.711 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.900832+0000", 2020-03-09T00:25:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.00062496000000000003 2020-03-09T00:25:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.712 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.901457+0000", 2020-03-09T00:25:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.3224000000000001e-05 2020-03-09T00:25:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.901470+0000", 2020-03-09T00:25:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0054500290000000003 2020-03-09T00:25:54.713 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.906921+0000", 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.3346999999999997e-05 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.714 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.906964+0000", 2020-03-09T00:25:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063740280000000003 2020-03-09T00:25:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.715 INFO:teuthology.orchestra.run.smithi104.stdout: "description": "MOSDECSubOpWrite(3.5s4 94/67 ECSubWrite(tid=61, reqid=client.4751.0:920, at_version=94'37, trim_to=0'0, roll_forward_to=87'35))", 2020-03-09T00:25:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "initiated_at": "2020-03-09T00:25:43.952200+0000", 2020-03-09T00:25:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "age": 10.591411857000001, 2020-03-09T00:25:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063906270000000003, 2020-03-09T00:25:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "type_data": { 2020-03-09T00:25:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "flag_point": "reached pg", 2020-03-09T00:25:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: "events": [ 2020-03-09T00:25:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "initiated", 2020-03-09T00:25:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952200+0000", 2020-03-09T00:25:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0 2020-03-09T00:25:54.717 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "throttled", 2020-03-09T00:25:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952200+0000", 2020-03-09T00:25:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 6.7176999999999997e-05 2020-03-09T00:25:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.718 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "header_read", 2020-03-09T00:25:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952267+0000", 2020-03-09T00:25:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 2.8873e-05 2020-03-09T00:25:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "all_read", 2020-03-09T00:25:54.719 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952296+0000", 2020-03-09T00:25:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.5406000000000001e-05 2020-03-09T00:25:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "dispatched", 2020-03-09T00:25:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952312+0000", 2020-03-09T00:25:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 7.4909999999999999e-06 2020-03-09T00:25:54.720 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "queued_for_pg", 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952319+0000", 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 5.8241e-05 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "reached_pg", 2020-03-09T00:25:54.721 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952377+0000", 2020-03-09T00:25:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 1.2758000000000001e-05 2020-03-09T00:25:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_started", 2020-03-09T00:25:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.952390+0000", 2020-03-09T00:25:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.006152533 2020-03-09T00:25:54.722 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "sub_op_committed", 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.958543+0000", 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 4.8148e-05 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: { 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "event": "done", 2020-03-09T00:25:54.723 INFO:teuthology.orchestra.run.smithi104.stdout: "time": "2020-03-09T00:25:43.958591+0000", 2020-03-09T00:25:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: "duration": 0.0063906270000000003 2020-03-09T00:25:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: } 2020-03-09T00:25:54.724 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2020-03-09T00:25:54.724 INFO:teuthology.orchestra.run.smithi104.stdout:} 2020-03-09T00:25:54.726 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2020-03-09T00:25:54.863 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:54.863 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:54.863 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2020-03-09T00:25:54.863 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:54.872 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2020-03-09T00:25:54.909 INFO:teuthology.orchestra.run.smithi148.stderr:noscrub is unset 2020-03-09T00:25:54.920 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub 2020-03-09T00:25:54.998 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:54.998 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2020-03-09T00:25:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2020-03-09T00:25:54.999 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2020-03-09T00:25:54.999 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:55.007 INFO:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2020-03-09T00:25:55.133 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2020-03-09T00:25:55.133 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2020-03-09T00:25:55.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2020-03-09T00:25:55.134 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2020-03-09T00:25:55.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.134 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs3 53/48 ECSubWrite(tid=91, reqid=client.4751.0:344, at_version=53'40, trim_to=0'0, roll_forward_to=52'38))", 2020-03-09T00:25:55.134 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:55.134 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.279173966000002, 2020-03-09T00:25:55.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649, 2020-03-09T00:25:55.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:55.135 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:55.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.851339+0000", 2020-03-09T00:25:55.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022148699999999999 2020-03-09T00:25:55.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853554+0000", 2020-03-09T00:25:55.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7487e-05 2020-03-09T00:25:55.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853571+0000", 2020-03-09T00:25:55.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1065e-05 2020-03-09T00:25:55.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853582+0000", 2020-03-09T00:25:55.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7259999999999999e-06 2020-03-09T00:25:55.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853588+0000", 2020-03-09T00:25:55.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6578e-05 2020-03-09T00:25:55.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853634+0000", 2020-03-09T00:25:55.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1282e-05 2020-03-09T00:25:55.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.853646+0000", 2020-03-09T00:25:55.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012718002000000001 2020-03-09T00:25:55.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866364+0000", 2020-03-09T00:25:55.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6638999999999999e-05 2020-03-09T00:25:55.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:02.866400+0000", 2020-03-09T00:25:55.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015061649 2020-03-09T00:25:55.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.143 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.144 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:375 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:55.144 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:55.144 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.329725951999997, 2020-03-09T00:25:55.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317, 2020-03-09T00:25:55.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.145 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.145 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.145 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.145 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.145 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 375 2020-03-09T00:25:55.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.145 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:55.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.800787+0000", 2020-03-09T00:25:55.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001381363 2020-03-09T00:25:55.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802168+0000", 2020-03-09T00:25:55.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5639000000000001e-05 2020-03-09T00:25:55.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802184+0000", 2020-03-09T00:25:55.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6752000000000001e-05 2020-03-09T00:25:55.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802240+0000", 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7064999999999999e-05 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802268+0000", 2020-03-09T00:25:55.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6162999999999996e-05 2020-03-09T00:25:55.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802344+0000", 2020-03-09T00:25:55.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000166458 2020-03-09T00:25:55.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.802510+0000", 2020-03-09T00:25:55.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018314410000000001 2020-03-09T00:25:55.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804342+0000", 2020-03-09T00:25:55.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079462270000000002 2020-03-09T00:25:55.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812288+0000", 2020-03-09T00:25:55.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078060580000000003 2020-03-09T00:25:55.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820094+0000", 2020-03-09T00:25:55.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110151 2020-03-09T00:25:55.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820204+0000", 2020-03-09T00:25:55.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019417317 2020-03-09T00:25:55.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.154 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.155 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:376 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~524288] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:55.155 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:55.155 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.326370617000002, 2020-03-09T00:25:55.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127, 2020-03-09T00:25:55.156 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.156 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.156 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.157 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.157 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 376 2020-03-09T00:25:55.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.157 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:55.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.804142+0000", 2020-03-09T00:25:55.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00133916 2020-03-09T00:25:55.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805481+0000", 2020-03-09T00:25:55.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1296e-05 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805492+0000", 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1435999999999999e-05 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805514+0000", 2020-03-09T00:25:55.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5969999999999998e-06 2020-03-09T00:25:55.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.805524+0000", 2020-03-09T00:25:55.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026245270000000002 2020-03-09T00:25:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808148+0000", 2020-03-09T00:25:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000157388 2020-03-09T00:25:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808305+0000", 2020-03-09T00:25:55.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00107099 2020-03-09T00:25:55.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.809376+0000", 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071133840000000004 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816490+0000", 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038222479999999999 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820312+0000", 2020-03-09T00:25:55.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012910100000000001 2020-03-09T00:25:55.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.820441+0000", 2020-03-09T00:25:55.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016299127 2020-03-09T00:25:55.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.166 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.166 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:377 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~245760] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:55.167 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:55.167 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.322511204999998, 2020-03-09T00:25:55.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858, 2020-03-09T00:25:55.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.168 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.169 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.169 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 377 2020-03-09T00:25:55.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.169 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808001+0000", 2020-03-09T00:25:55.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00047463399999999998 2020-03-09T00:25:55.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808476+0000", 2020-03-09T00:25:55.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7579999999999998e-06 2020-03-09T00:25:55.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808482+0000", 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7729999999999994e-06 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808492+0000", 2020-03-09T00:25:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808496+0000", 2020-03-09T00:25:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003930813 2020-03-09T00:25:55.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812427+0000", 2020-03-09T00:25:55.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013734399999999999 2020-03-09T00:25:55.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.812564+0000", 2020-03-09T00:25:55.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00094040499999999995 2020-03-09T00:25:55.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.813505+0000", 2020-03-09T00:25:55.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030313029999999999 2020-03-09T00:25:55.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816536+0000", 2020-03-09T00:25:55.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048022810000000003 2020-03-09T00:25:55.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821338+0000", 2020-03-09T00:25:55.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4952999999999997e-05 2020-03-09T00:25:55.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821433+0000", 2020-03-09T00:25:55.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013431858 2020-03-09T00:25:55.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.181 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.181 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:378 3.2s0 3:482d6e06:::smithi10428818-17:head [setxattr _header (57) in=7b] snapc 13=[13,11,10,f,c,5] ondisk+write+known_if_redirected e54)", 2020-03-09T00:25:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.321983080999999, 2020-03-09T00:25:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001, 2020-03-09T00:25:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.182 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.183 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.183 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 378 2020-03-09T00:25:55.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.183 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808530+0000", 2020-03-09T00:25:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2327999999999999e-05 2020-03-09T00:25:55.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808572+0000", 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7400000000000002e-06 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808576+0000", 2020-03-09T00:25:55.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5849999999999996e-06 2020-03-09T00:25:55.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808582+0000", 2020-03-09T00:25:55.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7070000000000001e-06 2020-03-09T00:25:55.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808585+0000", 2020-03-09T00:25:55.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068015719999999997 2020-03-09T00:25:55.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815386+0000", 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124704 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.815511+0000", 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00063270399999999999 2020-03-09T00:25:55.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816144+0000", 2020-03-09T00:25:55.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056543100000000003 2020-03-09T00:25:55.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816709+0000", 2020-03-09T00:25:55.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048141440000000002 2020-03-09T00:25:55.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821523+0000", 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012666000000000001 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.821650+0000", 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013120575000000001 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.191 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.192 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.192 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:379 3.2s0 3:482d6e06:::smithi10428818-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e54)", 2020-03-09T00:25:55.192 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:55.192 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.321903061, 2020-03-09T00:25:55.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999, 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 379 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.193 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808610+0000", 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3624000000000002e-05 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808643+0000", 2020-03-09T00:25:55.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.625e-06 2020-03-09T00:25:55.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808647+0000", 2020-03-09T00:25:55.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1500000000000002e-06 2020-03-09T00:25:55.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808654+0000", 2020-03-09T00:25:55.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8480000000000001e-06 2020-03-09T00:25:55.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.808657+0000", 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079120670000000001 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816569+0000", 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012729000000000001 2020-03-09T00:25:55.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:55.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.816696+0000", 2020-03-09T00:25:55.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054928110000000002 2020-03-09T00:25:55.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822189+0000", 2020-03-09T00:25:55.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6662000000000005e-05 2020-03-09T00:25:55.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.822276+0000", 2020-03-09T00:25:55.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001870029 2020-03-09T00:25:55.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:03.824146+0000", 2020-03-09T00:25:55.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015536105999999999 2020-03-09T00:25:55.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.203 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.203 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.203 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:505 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~0] snapc 15=[15,14,13,c,5] ondisk+write+known_if_redirected e59)", 2020-03-09T00:25:55.203 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:55.204 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.296303637999998, 2020-03-09T00:25:55.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016627194000000001, 2020-03-09T00:25:55.204 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.204 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.204 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.205 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.205 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.205 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 505 2020-03-09T00:25:55.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.205 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:55.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834209+0000", 2020-03-09T00:25:55.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7997000000000002e-05 2020-03-09T00:25:55.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834247+0000", 2020-03-09T00:25:55.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000001e-06 2020-03-09T00:25:55.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834251+0000", 2020-03-09T00:25:55.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.469e-06 2020-03-09T00:25:55.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834258+0000", 2020-03-09T00:25:55.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2660000000000002e-06 2020-03-09T00:25:55.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.834262+0000", 2020-03-09T00:25:55.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032379409999999998 2020-03-09T00:25:55.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837500+0000", 2020-03-09T00:25:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013429800000000001 2020-03-09T00:25:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.837634+0000", 2020-03-09T00:25:55.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076062099999999995 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838394+0000", 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039555900000000003 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.838790+0000", 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01196025 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.850750+0000", 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5890999999999995e-05 2020-03-09T00:25:55.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:08.850836+0000", 2020-03-09T00:25:55.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016627194000000001 2020-03-09T00:25:55.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.217 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.217 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:587 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~589824] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.154361762000001, 2020-03-09T00:25:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015855035, 2020-03-09T00:25:55.218 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 587 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.976151+0000", 2020-03-09T00:25:55.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001387634 2020-03-09T00:25:55.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977539+0000", 2020-03-09T00:25:55.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3190000000000001e-06 2020-03-09T00:25:55.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977548+0000", 2020-03-09T00:25:55.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5709000000000001e-05 2020-03-09T00:25:55.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977574+0000", 2020-03-09T00:25:55.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.471e-05 2020-03-09T00:25:55.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977588+0000", 2020-03-09T00:25:55.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8529000000000006e-05 2020-03-09T00:25:55.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977677+0000", 2020-03-09T00:25:55.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013839599999999999 2020-03-09T00:25:55.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.977815+0000", 2020-03-09T00:25:55.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015356079999999999 2020-03-09T00:25:55.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979351+0000", 2020-03-09T00:25:55.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070753639999999998 2020-03-09T00:25:55.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986426+0000", 2020-03-09T00:25:55.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054801509999999999 2020-03-09T00:25:55.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991906+0000", 2020-03-09T00:25:55.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9615000000000001e-05 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992006+0000", 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015855035 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.231 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:588 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~540672] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:55.231 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:55.231 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.151304074000002, 2020-03-09T00:25:55.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013267434999999999, 2020-03-09T00:25:55.231 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.232 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.232 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.232 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.232 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.232 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 588 2020-03-09T00:25:55.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.232 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:55.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.979209+0000", 2020-03-09T00:25:55.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001696822 2020-03-09T00:25:55.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980905+0000", 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7710000000000006e-06 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980913+0000", 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4919999999999999e-05 2020-03-09T00:25:55.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980928+0000", 2020-03-09T00:25:55.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.34e-06 2020-03-09T00:25:55.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.980935+0000", 2020-03-09T00:25:55.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015720580000000001 2020-03-09T00:25:55.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.982507+0000", 2020-03-09T00:25:55.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7051000000000004e-05 2020-03-09T00:25:55.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.982595+0000", 2020-03-09T00:25:55.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081195499999999995 2020-03-09T00:25:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983406+0000", 2020-03-09T00:25:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082914860000000007 2020-03-09T00:25:55.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991698+0000", 2020-03-09T00:25:55.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058976000000000005 2020-03-09T00:25:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992288+0000", 2020-03-09T00:25:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018827199999999999 2020-03-09T00:25:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992476+0000", 2020-03-09T00:25:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013267434999999999 2020-03-09T00:25:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:589 3.2s0 3:49883c8e:::smithi10428818-40:head [append 0~622592] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.146957170999997, 2020-03-09T00:25:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012556799, 2020-03-09T00:25:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.244 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.245 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.245 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.245 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.245 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 589 2020-03-09T00:25:55.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.245 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:55.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.983555+0000", 2020-03-09T00:25:55.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019622569999999998 2020-03-09T00:25:55.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985518+0000", 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3129999999999999e-06 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985526+0000", 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4384e-05 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985540+0000", 2020-03-09T00:25:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2789999999999998e-06 2020-03-09T00:25:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985549+0000", 2020-03-09T00:25:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097411399999999999 2020-03-09T00:25:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986523+0000", 2020-03-09T00:25:55.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5084000000000001e-05 2020-03-09T00:25:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.986598+0000", 2020-03-09T00:25:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010282659999999999 2020-03-09T00:25:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.987626+0000", 2020-03-09T00:25:55.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040731630000000003 2020-03-09T00:25:55.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991699+0000", 2020-03-09T00:25:55.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043120320000000004 2020-03-09T00:25:55.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.253 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.253 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996011+0000", 2020-03-09T00:25:55.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100907 2020-03-09T00:25:55.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996112+0000", 2020-03-09T00:25:55.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012556799 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:590 3.2s0 3:49883c8e:::smithi10428818-40:head [setxattr _header (57) in=7b] snapc 1f=[1f,1a,13] ondisk+write+known_if_redirected e68)", 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:55.255 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.144903771999999, 2020-03-09T00:25:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010789646, 2020-03-09T00:25:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.256 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 590 2020-03-09T00:25:55.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.257 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:55.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985609+0000", 2020-03-09T00:25:55.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2869999999999998e-05 2020-03-09T00:25:55.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985692+0000", 2020-03-09T00:25:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.507e-06 2020-03-09T00:25:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985700+0000", 2020-03-09T00:25:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5087e-05 2020-03-09T00:25:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985715+0000", 2020-03-09T00:25:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5260000000000001e-06 2020-03-09T00:25:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985721+0000", 2020-03-09T00:25:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0051320369999999999 2020-03-09T00:25:55.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.990853+0000", 2020-03-09T00:25:55.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114726 2020-03-09T00:25:55.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.990968+0000", 2020-03-09T00:25:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000440037 2020-03-09T00:25:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.991408+0000", 2020-03-09T00:25:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001238942 2020-03-09T00:25:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992647+0000", 2020-03-09T00:25:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036263749999999998 2020-03-09T00:25:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996273+0000", 2020-03-09T00:25:55.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012553900000000001 2020-03-09T00:25:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.996399+0000", 2020-03-09T00:25:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010789646 2020-03-09T00:25:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.266 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.267 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.267 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:591 3.2s0 3:49883c8e:::smithi10428818-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e68)", 2020-03-09T00:25:55.267 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:55.267 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.144744635000002, 2020-03-09T00:25:55.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015712046, 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 591 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.268 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:55.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985768+0000", 2020-03-09T00:25:55.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3307999999999999e-05 2020-03-09T00:25:55.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985821+0000", 2020-03-09T00:25:55.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5400000000000001e-06 2020-03-09T00:25:55.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985828+0000", 2020-03-09T00:25:55.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2712e-05 2020-03-09T00:25:55.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985841+0000", 2020-03-09T00:25:55.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6920000000000001e-06 2020-03-09T00:25:55.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.985845+0000", 2020-03-09T00:25:55.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066716329999999997 2020-03-09T00:25:55.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992517+0000", 2020-03-09T00:25:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012270799999999999 2020-03-09T00:25:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.992640+0000", 2020-03-09T00:25:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00472662 2020-03-09T00:25:55.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.997366+0000", 2020-03-09T00:25:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012992200000000001 2020-03-09T00:25:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:17.997496+0000", 2020-03-09T00:25:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039839109999999997 2020-03-09T00:25:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:18.001480+0000", 2020-03-09T00:25:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015712046 2020-03-09T00:25:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.6s6 80/67 ECSubWrite(tid=116, reqid=client.4751.0:672, at_version=80'33, trim_to=0'0, roll_forward_to=76'31))", 2020-03-09T00:25:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:29.821736+0000", 2020-03-09T00:25:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.308776437999999, 2020-03-09T00:25:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013523835, 2020-03-09T00:25:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821736+0000", 2020-03-09T00:25:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.821736+0000", 2020-03-09T00:25:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011537120000000001 2020-03-09T00:25:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822890+0000", 2020-03-09T00:25:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3646000000000003e-05 2020-03-09T00:25:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822924+0000", 2020-03-09T00:25:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9941000000000001e-05 2020-03-09T00:25:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822943+0000", 2020-03-09T00:25:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0580000000000004e-06 2020-03-09T00:25:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.822953+0000", 2020-03-09T00:25:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4598999999999996e-05 2020-03-09T00:25:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.823017+0000", 2020-03-09T00:25:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3725e-05 2020-03-09T00:25:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.823031+0000", 2020-03-09T00:25:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012191274 2020-03-09T00:25:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.835222+0000", 2020-03-09T00:25:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7880000000000003e-05 2020-03-09T00:25:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:29.835260+0000", 2020-03-09T00:25:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013523835 2020-03-09T00:25:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.1s2 85/67 ECSubWrite(tid=242, reqid=client.4751.0:758, at_version=85'73, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:34.885073+0000", 2020-03-09T00:25:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.245439366999999, 2020-03-09T00:25:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015418995, 2020-03-09T00:25:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2020-03-09T00:25:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.885073+0000", 2020-03-09T00:25:55.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.885073+0000", 2020-03-09T00:25:55.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027330150000000001 2020-03-09T00:25:55.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887806+0000", 2020-03-09T00:25:55.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1937999999999998e-05 2020-03-09T00:25:55.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887838+0000", 2020-03-09T00:25:55.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7982000000000001e-05 2020-03-09T00:25:55.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887866+0000", 2020-03-09T00:25:55.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1049999999999997e-06 2020-03-09T00:25:55.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887873+0000", 2020-03-09T00:25:55.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7388e-05 2020-03-09T00:25:55.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887931+0000", 2020-03-09T00:25:55.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2594e-05 2020-03-09T00:25:55.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.887943+0000", 2020-03-09T00:25:55.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01248904 2020-03-09T00:25:55.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900432+0000", 2020-03-09T00:25:55.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9932999999999997e-05 2020-03-09T00:25:55.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:34.900492+0000", 2020-03-09T00:25:55.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015418995 2020-03-09T00:25:55.297 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.297 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.297 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.297 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:868 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~753664] snapc 35=[] ondisk+write+known_if_redirected e91)", 2020-03-09T00:25:55.298 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:40.851501+0000", 2020-03-09T00:25:55.298 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.279011761, 2020-03-09T00:25:55.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014552036000000001, 2020-03-09T00:25:55.298 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.298 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.298 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.298 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.299 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.299 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 868 2020-03-09T00:25:55.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.299 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.851501+0000", 2020-03-09T00:25:55.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.851501+0000", 2020-03-09T00:25:55.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089851500000000001 2020-03-09T00:25:55.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.852399+0000", 2020-03-09T00:25:55.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9860000000000006e-06 2020-03-09T00:25:55.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.302 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.302 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.852409+0000", 2020-03-09T00:25:55.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.419e-05 2020-03-09T00:25:55.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.852424+0000", 2020-03-09T00:25:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.687e-06 2020-03-09T00:25:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.852432+0000", 2020-03-09T00:25:55.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1569999999999995e-05 2020-03-09T00:25:55.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.852494+0000", 2020-03-09T00:25:55.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115177 2020-03-09T00:25:55.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.852609+0000", 2020-03-09T00:25:55.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001347509 2020-03-09T00:25:55.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.853957+0000", 2020-03-09T00:25:55.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038933040000000002 2020-03-09T00:25:55.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.857850+0000", 2020-03-09T00:25:55.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080775489999999998 2020-03-09T00:25:55.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.865927+0000", 2020-03-09T00:25:55.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000125549 2020-03-09T00:25:55.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.866053+0000", 2020-03-09T00:25:55.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014552036000000001 2020-03-09T00:25:55.309 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.309 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.309 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.309 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:869 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~786432] snapc 35=[] ondisk+write+known_if_redirected e91)", 2020-03-09T00:25:55.309 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:40.857949+0000", 2020-03-09T00:25:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.272564131999999, 2020-03-09T00:25:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012660891000000001, 2020-03-09T00:25:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.310 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.311 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 869 2020-03-09T00:25:55.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.311 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.857949+0000", 2020-03-09T00:25:55.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.857949+0000", 2020-03-09T00:25:55.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082880599999999997 2020-03-09T00:25:55.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.858777+0000", 2020-03-09T00:25:55.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2540000000000003e-06 2020-03-09T00:25:55.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.858785+0000", 2020-03-09T00:25:55.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0701e-05 2020-03-09T00:25:55.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.858795+0000", 2020-03-09T00:25:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0979999999999999e-06 2020-03-09T00:25:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.858801+0000", 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5424999999999999e-05 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.858857+0000", 2020-03-09T00:25:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101103 2020-03-09T00:25:55.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.858958+0000", 2020-03-09T00:25:55.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012048880000000001 2020-03-09T00:25:55.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.860163+0000", 2020-03-09T00:25:55.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054948699999999998 2020-03-09T00:25:55.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.865658+0000", 2020-03-09T00:25:55.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048288189999999998 2020-03-09T00:25:55.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.870486+0000", 2020-03-09T00:25:55.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000122927 2020-03-09T00:25:55.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.870609+0000", 2020-03-09T00:25:55.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012660891000000001 2020-03-09T00:25:55.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.320 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.321 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.321 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:870 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~655360] snapc 35=[] ondisk+write+known_if_redirected e91)", 2020-03-09T00:25:55.321 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:40.860837+0000", 2020-03-09T00:25:55.321 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.269675791999999, 2020-03-09T00:25:55.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013019162000000001, 2020-03-09T00:25:55.322 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.322 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.322 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.322 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.322 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.323 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 870 2020-03-09T00:25:55.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.323 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.860837+0000", 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.860837+0000", 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00085439999999999995 2020-03-09T00:25:55.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861691+0000", 2020-03-09T00:25:55.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4590000000000003e-06 2020-03-09T00:25:55.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861698+0000", 2020-03-09T00:25:55.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4329999999999993e-06 2020-03-09T00:25:55.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861707+0000", 2020-03-09T00:25:55.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0479999999999998e-06 2020-03-09T00:25:55.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861712+0000", 2020-03-09T00:25:55.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004717995 2020-03-09T00:25:55.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.866430+0000", 2020-03-09T00:25:55.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104473 2020-03-09T00:25:55.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.866535+0000", 2020-03-09T00:25:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00094798599999999995 2020-03-09T00:25:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.867483+0000", 2020-03-09T00:25:55.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056295529999999998 2020-03-09T00:25:55.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.873112+0000", 2020-03-09T00:25:55.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062808299999999998 2020-03-09T00:25:55.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.873740+0000", 2020-03-09T00:25:55.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115732 2020-03-09T00:25:55.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.873856+0000", 2020-03-09T00:25:55.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013019162000000001 2020-03-09T00:25:55.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.333 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:871 3.2s0 3:482d6e06:::smithi10428818-17:head [append 0~114688] snapc 35=[] ondisk+write+known_if_redirected e91)", 2020-03-09T00:25:55.333 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:40.861745+0000", 2020-03-09T00:25:55.334 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.268767903000001, 2020-03-09T00:25:55.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012421463000000001, 2020-03-09T00:25:55.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.335 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.335 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.335 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.335 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 871 2020-03-09T00:25:55.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.336 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861745+0000", 2020-03-09T00:25:55.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861745+0000", 2020-03-09T00:25:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115149 2020-03-09T00:25:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861860+0000", 2020-03-09T00:25:55.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2259999999999996e-06 2020-03-09T00:25:55.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861865+0000", 2020-03-09T00:25:55.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.04e-05 2020-03-09T00:25:55.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861876+0000", 2020-03-09T00:25:55.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.177e-06 2020-03-09T00:25:55.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861880+0000", 2020-03-09T00:25:55.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088069940000000003 2020-03-09T00:25:55.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.870687+0000", 2020-03-09T00:25:55.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5495999999999995e-05 2020-03-09T00:25:55.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.870782+0000", 2020-03-09T00:25:55.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062034799999999995 2020-03-09T00:25:55.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.871403+0000", 2020-03-09T00:25:55.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018332380000000001 2020-03-09T00:25:55.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.873236+0000", 2020-03-09T00:25:55.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083644899999999996 2020-03-09T00:25:55.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.874072+0000", 2020-03-09T00:25:55.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3986000000000004e-05 2020-03-09T00:25:55.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.874166+0000", 2020-03-09T00:25:55.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012421463000000001 2020-03-09T00:25:55.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.346 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.346 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:872 3.2s0 3:482d6e06:::smithi10428818-17:head [setxattr _header (57) in=7b] snapc 35=[] ondisk+write+known_if_redirected e91)", 2020-03-09T00:25:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:40.861913+0000", 2020-03-09T00:25:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.268600072, 2020-03-09T00:25:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012668749999999999, 2020-03-09T00:25:55.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 872 2020-03-09T00:25:55.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861913+0000", 2020-03-09T00:25:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861913+0000", 2020-03-09T00:25:55.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5061e-05 2020-03-09T00:25:55.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861968+0000", 2020-03-09T00:25:55.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6099999999999999e-06 2020-03-09T00:25:55.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861972+0000", 2020-03-09T00:25:55.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2919999999999998e-06 2020-03-09T00:25:55.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861980+0000", 2020-03-09T00:25:55.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1279999999999997e-06 2020-03-09T00:25:55.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.861989+0000", 2020-03-09T00:25:55.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010146486999999999 2020-03-09T00:25:55.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.872135+0000", 2020-03-09T00:25:55.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102237 2020-03-09T00:25:55.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.872237+0000", 2020-03-09T00:25:55.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062878499999999998 2020-03-09T00:25:55.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.872866+0000", 2020-03-09T00:25:55.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000453516 2020-03-09T00:25:55.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.873320+0000", 2020-03-09T00:25:55.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011609000000000001 2020-03-09T00:25:55.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2020-03-09T00:25:55.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.874481+0000", 2020-03-09T00:25:55.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100734 2020-03-09T00:25:55.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.874581+0000", 2020-03-09T00:25:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012668749999999999 2020-03-09T00:25:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.360 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4751.0:873 3.2s0 3:482d6e06:::smithi10428818-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e91)", 2020-03-09T00:25:55.360 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2020-03-09T00:25:40.862017+0000", 2020-03-09T00:25:55.360 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.268495377000001, 2020-03-09T00:25:55.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014824117, 2020-03-09T00:25:55.360 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2020-03-09T00:25:55.360 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2020-03-09T00:25:55.360 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2020-03-09T00:25:55.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4751", 2020-03-09T00:25:55.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.361 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 873 2020-03-09T00:25:55.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.361 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2020-03-09T00:25:55.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2020-03-09T00:25:55.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.862017+0000", 2020-03-09T00:25:55.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2020-03-09T00:25:55.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2020-03-09T00:25:55.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.862017+0000", 2020-03-09T00:25:55.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4476000000000001e-05 2020-03-09T00:25:55.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2020-03-09T00:25:55.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.862042+0000", 2020-03-09T00:25:55.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.517e-06 2020-03-09T00:25:55.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2020-03-09T00:25:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.862045+0000", 2020-03-09T00:25:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1520000000000003e-06 2020-03-09T00:25:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2020-03-09T00:25:55.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.862052+0000", 2020-03-09T00:25:55.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.853e-06 2020-03-09T00:25:55.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.862055+0000", 2020-03-09T00:25:55.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013068929999999999 2020-03-09T00:25:55.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2020-03-09T00:25:55.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.875124+0000", 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014560699999999999 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.875270+0000", 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001571582 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2020-03-09T00:25:55.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2020-03-09T00:25:55.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2020-03-09T00:25:55.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2020-03-09T00:25:40.876841+0000", 2020-03-09T00:25:55.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014824117 2020-03-09T00:25:55.368 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.368 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2020-03-09T00:25:55.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2020-03-09T00:25:55.369 INFO:teuthology.orchestra.run.smithi050.stdout:} 2020-03-09T00:25:55.372 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2020-03-09T00:25:55.488 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:55.488 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:55.488 INFO:teuthology.orchestra.run.smithi148.stdout: "num_ops": 0 2020-03-09T00:25:55.488 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:55.497 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2020-03-09T00:25:55.614 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:55.614 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [], 2020-03-09T00:25:55.615 INFO:teuthology.orchestra.run.smithi148.stdout: "complaint_time": 30, 2020-03-09T00:25:55.615 INFO:teuthology.orchestra.run.smithi148.stdout: "num_blocked_ops": 0 2020-03-09T00:25:55.615 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:55.623 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2020-03-09T00:25:55.740 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2020-03-09T00:25:55.740 INFO:teuthology.orchestra.run.smithi148.stdout: "size": 20, 2020-03-09T00:25:55.741 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 600, 2020-03-09T00:25:55.741 INFO:teuthology.orchestra.run.smithi148.stdout: "ops": [ 2020-03-09T00:25:55.741 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.741 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:242 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~638976] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:55.741 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 61.891047188000002, 2020-03-09T00:25:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001, 2020-03-09T00:25:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.743 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 242 2020-03-09T00:25:55.743 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.743 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.743 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.743 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.743 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:55.743 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.846408+0000", 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012420980000000001 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.744 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.745 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847650+0000", 2020-03-09T00:25:55.745 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2689999999999996e-06 2020-03-09T00:25:55.745 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.745 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.745 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.745 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847657+0000", 2020-03-09T00:25:55.745 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2094e-05 2020-03-09T00:25:55.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.746 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.746 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847669+0000", 2020-03-09T00:25:55.746 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2690000000000002e-06 2020-03-09T00:25:55.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.747 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847676+0000", 2020-03-09T00:25:55.747 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.3765999999999999e-05 2020-03-09T00:25:55.747 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.747 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.747 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847740+0000", 2020-03-09T00:25:55.748 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000155716 2020-03-09T00:25:55.748 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.748 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.748 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.748 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.847896+0000", 2020-03-09T00:25:55.749 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0020523669999999998 2020-03-09T00:25:55.749 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.749 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.749 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.749 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.849948+0000", 2020-03-09T00:25:55.749 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010549378999999999 2020-03-09T00:25:55.750 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.750 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.750 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.750 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860497+0000", 2020-03-09T00:25:55.750 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046388970000000003 2020-03-09T00:25:55.751 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.751 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.751 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.751 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865136+0000", 2020-03-09T00:25:55.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.2514000000000005e-05 2020-03-09T00:25:55.752 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.752 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.752 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.752 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865199+0000", 2020-03-09T00:25:55.752 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018790369000000001 2020-03-09T00:25:55.753 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.753 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.753 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.753 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.753 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.753 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:243 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~376832] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:55.753 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:55.754 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 61.888541385000003, 2020-03-09T00:25:55.754 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999, 2020-03-09T00:25:55.754 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.754 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.754 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.754 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.754 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.755 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 243 2020-03-09T00:25:55.755 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.755 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.755 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.755 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.755 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:55.755 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.756 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.756 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.756 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.848914+0000", 2020-03-09T00:25:55.756 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0012192539999999999 2020-03-09T00:25:55.756 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.756 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.757 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.757 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850133+0000", 2020-03-09T00:25:55.757 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1730000000000001e-06 2020-03-09T00:25:55.757 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.757 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.757 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850143+0000", 2020-03-09T00:25:55.758 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.1009000000000001e-05 2020-03-09T00:25:55.758 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.758 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.758 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.758 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850174+0000", 2020-03-09T00:25:55.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.0500000000000003e-06 2020-03-09T00:25:55.759 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.759 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.759 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.759 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850181+0000", 2020-03-09T00:25:55.759 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047873150000000003 2020-03-09T00:25:55.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.760 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.760 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.760 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.854968+0000", 2020-03-09T00:25:55.760 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013640000000000001 2020-03-09T00:25:55.760 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.761 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.761 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.855104+0000", 2020-03-09T00:25:55.761 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001096669 2020-03-09T00:25:55.761 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.761 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.762 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.856201+0000", 2020-03-09T00:25:55.762 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0042977170000000004 2020-03-09T00:25:55.762 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.762 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.762 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.763 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.860499+0000", 2020-03-09T00:25:55.763 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.008534827 2020-03-09T00:25:55.763 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.763 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.763 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.763 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869034+0000", 2020-03-09T00:25:55.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4033e-05 2020-03-09T00:25:55.764 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.764 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.764 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.764 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869118+0000", 2020-03-09T00:25:55.764 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020203446999999999 2020-03-09T00:25:55.764 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.765 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.765 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.765 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.765 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.765 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:244 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (56) in=7b] snapc 9=[9,5,4] ondisk+write+known_if_redirected e44)", 2020-03-09T00:25:55.765 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:55.765 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 61.887196166000003, 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999, 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 244 2020-03-09T00:25:55.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.767 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.767 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.767 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.767 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:55.767 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.768 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.768 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850259+0000", 2020-03-09T00:25:55.768 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000146107 2020-03-09T00:25:55.768 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.768 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.769 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.769 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850405+0000", 2020-03-09T00:25:55.769 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2099999999999999e-05 2020-03-09T00:25:55.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850418+0000", 2020-03-09T00:25:55.770 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9311000000000001e-05 2020-03-09T00:25:55.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.770 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.770 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850447+0000", 2020-03-09T00:25:55.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6460000000000004e-06 2020-03-09T00:25:55.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.771 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.771 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850453+0000", 2020-03-09T00:25:55.771 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011322660999999999 2020-03-09T00:25:55.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.772 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.772 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861776+0000", 2020-03-09T00:25:55.772 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00015394900000000001 2020-03-09T00:25:55.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.773 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.861930+0000", 2020-03-09T00:25:55.773 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00076229599999999998 2020-03-09T00:25:55.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.773 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.773 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.862692+0000", 2020-03-09T00:25:55.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002593092 2020-03-09T00:25:55.774 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.774 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.774 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.865285+0000", 2020-03-09T00:25:55.774 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0039673670000000003 2020-03-09T00:25:55.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.775 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.775 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869253+0000", 2020-03-09T00:25:55.775 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00010661 2020-03-09T00:25:55.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.776 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.776 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869359+0000", 2020-03-09T00:25:55.776 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019100138999999999 2020-03-09T00:25:55.776 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.776 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.776 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.777 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:245 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-03-09T00:25:55.777 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:55.777 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 61.886949930999997, 2020-03-09T00:25:55.777 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998, 2020-03-09T00:25:55.777 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.777 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:55.778 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.778 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.778 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.778 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 245 2020-03-09T00:25:55.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.778 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.779 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:55.779 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.779 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.779 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850506+0000", 2020-03-09T00:25:55.780 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2926999999999997e-05 2020-03-09T00:25:55.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.780 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.780 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850548+0000", 2020-03-09T00:25:55.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7159999999999997e-06 2020-03-09T00:25:55.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.781 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.781 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850555+0000", 2020-03-09T00:25:55.781 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2537999999999999e-05 2020-03-09T00:25:55.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.782 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.782 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850568+0000", 2020-03-09T00:25:55.782 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6200000000000004e-06 2020-03-09T00:25:55.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.783 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.783 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.850573+0000", 2020-03-09T00:25:55.784 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.013277631 2020-03-09T00:25:55.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.785 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.785 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.863851+0000", 2020-03-09T00:25:55.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00016975200000000001 2020-03-09T00:25:55.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.786 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:55.786 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.864021+0000", 2020-03-09T00:25:55.786 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0058362869999999999 2020-03-09T00:25:55.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.787 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.787 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869857+0000", 2020-03-09T00:25:55.787 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00011197 2020-03-09T00:25:55.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.869969+0000", 2020-03-09T00:25:55.788 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0016289570000000001 2020-03-09T00:25:55.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.788 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.788 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:53.871598+0000", 2020-03-09T00:25:55.789 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021092397999999998 2020-03-09T00:25:55.789 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.789 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.789 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.789 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.789 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:289 3.es0 3:73be088e:::smithi10428818-42:head [append 0~737280] snapc f=[f,c,5,4] ondisk+write+known_if_redirected e49)", 2020-03-09T00:25:55.790 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:55.790 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 56.922977695, 2020-03-09T00:25:55.790 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001, 2020-03-09T00:25:55.790 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.790 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.791 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.791 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.791 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.791 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 289 2020-03-09T00:25:55.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.792 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.792 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.792 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.792 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:55.792 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.793 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.793 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.814478+0000", 2020-03-09T00:25:55.793 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001180787 2020-03-09T00:25:55.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.793 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.794 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.794 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815659+0000", 2020-03-09T00:25:55.794 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5960000000000004e-06 2020-03-09T00:25:55.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.795 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.795 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.795 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815667+0000", 2020-03-09T00:25:55.795 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.7425e-05 2020-03-09T00:25:55.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.796 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.796 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.797 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815685+0000", 2020-03-09T00:25:55.797 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0162e-05 2020-03-09T00:25:55.797 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.797 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815695+0000", 2020-03-09T00:25:55.798 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2490000000000004e-05 2020-03-09T00:25:55.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.798 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.798 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815787+0000", 2020-03-09T00:25:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7851999999999998e-05 2020-03-09T00:25:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.815885+0000", 2020-03-09T00:25:55.799 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001491497 2020-03-09T00:25:55.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.800 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.800 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817377+0000", 2020-03-09T00:25:55.801 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010162513999999999 2020-03-09T00:25:55.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.801 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.827539+0000", 2020-03-09T00:25:55.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0027651400000000001 2020-03-09T00:25:55.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.802 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.802 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830304+0000", 2020-03-09T00:25:55.802 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.8808999999999995e-05 2020-03-09T00:25:55.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.803 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.803 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830403+0000", 2020-03-09T00:25:55.803 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.015925272000000001 2020-03-09T00:25:55.804 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.804 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.804 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.804 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:292 3.es0 3:73be088e:::smithi10428818-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e49)", 2020-03-09T00:25:55.804 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:55.805 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 56.919674671999999, 2020-03-09T00:25:55.805 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999, 2020-03-09T00:25:55.805 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.805 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:55.805 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.806 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.806 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.806 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 292 2020-03-09T00:25:55.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.806 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.807 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.807 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:55.807 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.807 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.808 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817781+0000", 2020-03-09T00:25:55.808 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7518000000000001e-05 2020-03-09T00:25:55.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.808 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817828+0000", 2020-03-09T00:25:55.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.1009999999999997e-06 2020-03-09T00:25:55.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.809 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.809 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817834+0000", 2020-03-09T00:25:55.809 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2721000000000001e-05 2020-03-09T00:25:55.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.810 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.810 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817847+0000", 2020-03-09T00:25:55.810 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2339999999999996e-06 2020-03-09T00:25:55.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.811 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.811 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.817851+0000", 2020-03-09T00:25:55.811 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.012178102999999999 2020-03-09T00:25:55.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.812 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830029+0000", 2020-03-09T00:25:55.812 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012795100000000001 2020-03-09T00:25:55.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.812 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:55.813 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.830157+0000", 2020-03-09T00:25:55.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003501414 2020-03-09T00:25:55.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.813 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.813 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833659+0000", 2020-03-09T00:25:55.813 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5081999999999996e-05 2020-03-09T00:25:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.833744+0000", 2020-03-09T00:25:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002393077 2020-03-09T00:25:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.815 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.815 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:24:58.836137+0000", 2020-03-09T00:25:55.815 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018356200999999999 2020-03-09T00:25:55.815 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.815 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.815 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.816 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:339 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~475136] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:55.816 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:55.816 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 52.929767101000003, 2020-03-09T00:25:55.816 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998, 2020-03-09T00:25:55.817 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.817 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.817 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.817 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.817 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.817 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 339 2020-03-09T00:25:55.818 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.818 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.818 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.818 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:55.818 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.807688+0000", 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00074630200000000001 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.819 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808435+0000", 2020-03-09T00:25:55.820 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.6087e-05 2020-03-09T00:25:55.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.820 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.820 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.820 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808451+0000", 2020-03-09T00:25:55.821 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.413e-05 2020-03-09T00:25:55.821 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.821 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.821 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.821 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808465+0000", 2020-03-09T00:25:55.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.4320000000000004e-06 2020-03-09T00:25:55.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.822 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.822 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.822 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.808471+0000", 2020-03-09T00:25:55.822 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0046408889999999996 2020-03-09T00:25:55.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813112+0000", 2020-03-09T00:25:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013929799999999999 2020-03-09T00:25:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.824 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.813252+0000", 2020-03-09T00:25:55.824 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0017257030000000001 2020-03-09T00:25:55.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.824 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.825 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.814977+0000", 2020-03-09T00:25:55.825 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0099845400000000001 2020-03-09T00:25:55.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.825 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.825 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.824962+0000", 2020-03-09T00:25:55.826 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00049262399999999999 2020-03-09T00:25:55.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.826 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.826 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825454+0000", 2020-03-09T00:25:55.826 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.462e-05 2020-03-09T00:25:55.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.827 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.827 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825539+0000", 2020-03-09T00:25:55.827 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.017850624999999998 2020-03-09T00:25:55.827 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.827 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.827 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.828 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:340 3.es0 3:7924a6e6:::smithi10428818-20:head [append 0~294912] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:55.828 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:55.828 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 52.927162672999998, 2020-03-09T00:25:55.829 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244, 2020-03-09T00:25:55.829 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.829 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.829 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.829 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.829 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.829 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 340 2020-03-09T00:25:55.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.830 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.830 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.830 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:55.830 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810293+0000", 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000216388 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.831 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810509+0000", 2020-03-09T00:25:55.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.6459999999999998e-06 2020-03-09T00:25:55.832 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.832 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.832 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.832 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810515+0000", 2020-03-09T00:25:55.832 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.799e-06 2020-03-09T00:25:55.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.833 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.833 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.833 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810524+0000", 2020-03-09T00:25:55.833 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.5940000000000004e-06 2020-03-09T00:25:55.833 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.834 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.834 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810528+0000", 2020-03-09T00:25:55.834 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.007805228 2020-03-09T00:25:55.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.834 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.835 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818333+0000", 2020-03-09T00:25:55.835 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013953500000000001 2020-03-09T00:25:55.835 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.835 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.835 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.818473+0000", 2020-03-09T00:25:55.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00094954999999999996 2020-03-09T00:25:55.836 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.836 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.836 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.836 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.819423+0000", 2020-03-09T00:25:55.836 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0056089679999999998 2020-03-09T00:25:55.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.837 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.837 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.825032+0000", 2020-03-09T00:25:55.837 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0015242280000000001 2020-03-09T00:25:55.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.838 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.838 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826556+0000", 2020-03-09T00:25:55.838 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4308000000000007e-05 2020-03-09T00:25:55.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.839 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.839 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.826640+0000", 2020-03-09T00:25:55.839 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016347244 2020-03-09T00:25:55.840 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.840 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.840 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.840 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:341 3.es0 3:7924a6e6:::smithi10428818-20:head [setxattr _header (57) in=7b] snapc 11=[11,10,f,c,5] ondisk+write+known_if_redirected e53)", 2020-03-09T00:25:55.841 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:55.841 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 52.926897590000003, 2020-03-09T00:25:55.841 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001, 2020-03-09T00:25:55.841 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.841 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.841 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.841 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.842 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.842 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 341 2020-03-09T00:25:55.842 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.842 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.843 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.843 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:55.843 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.844 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810558+0000", 2020-03-09T00:25:55.844 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.6538999999999997e-05 2020-03-09T00:25:55.844 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.844 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810594+0000", 2020-03-09T00:25:55.845 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.3629999999999998e-06 2020-03-09T00:25:55.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.845 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.845 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810598+0000", 2020-03-09T00:25:55.846 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1715e-05 2020-03-09T00:25:55.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.846 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.846 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810610+0000", 2020-03-09T00:25:55.847 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7019999999999999e-06 2020-03-09T00:25:55.847 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.847 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.847 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.847 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810613+0000", 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016583064000000002 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827196+0000", 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000134802 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.848 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.849 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.849 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.827331+0000", 2020-03-09T00:25:55.849 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00079633999999999998 2020-03-09T00:25:55.849 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.849 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.850 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.850 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828127+0000", 2020-03-09T00:25:55.850 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000914331 2020-03-09T00:25:55.850 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.850 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.850 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.851 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.829042+0000", 2020-03-09T00:25:55.851 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0045226199999999998 2020-03-09T00:25:55.851 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.851 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.851 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833564+0000", 2020-03-09T00:25:55.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000101851 2020-03-09T00:25:55.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.852 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.852 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.833666+0000", 2020-03-09T00:25:55.852 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.023108327000000001 2020-03-09T00:25:55.853 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.853 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.853 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.853 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:342 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-03-09T00:25:55.854 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:55.854 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 52.926819489000003, 2020-03-09T00:25:55.854 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001, 2020-03-09T00:25:55.854 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.854 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:55.854 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.854 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.855 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.855 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 342 2020-03-09T00:25:55.855 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.855 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.855 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.856 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:55.856 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.856 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.856 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.856 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.856 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810636+0000", 2020-03-09T00:25:55.856 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.1574e-05 2020-03-09T00:25:55.857 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.857 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.857 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.857 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810658+0000", 2020-03-09T00:25:55.857 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.196e-06 2020-03-09T00:25:55.857 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.858 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.858 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.858 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810661+0000", 2020-03-09T00:25:55.858 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.6000000000000003e-06 2020-03-09T00:25:55.858 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.858 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.858 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.859 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810667+0000", 2020-03-09T00:25:55.859 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3269999999999999e-06 2020-03-09T00:25:55.859 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.859 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.859 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.859 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.810670+0000", 2020-03-09T00:25:55.859 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018222015000000001 2020-03-09T00:25:55.860 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.860 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.860 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.860 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828892+0000", 2020-03-09T00:25:55.860 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000107623 2020-03-09T00:25:55.861 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.861 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.861 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:55.861 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.828999+0000", 2020-03-09T00:25:55.861 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.005589938 2020-03-09T00:25:55.862 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.862 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.862 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.862 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834589+0000", 2020-03-09T00:25:55.862 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012716299999999999 2020-03-09T00:25:55.862 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.862 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.863 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.863 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.834716+0000", 2020-03-09T00:25:55.863 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0048828580000000003 2020-03-09T00:25:55.863 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.863 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.863 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.864 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:02.839599+0000", 2020-03-09T00:25:55.864 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.028963294000000001 2020-03-09T00:25:55.864 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.864 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.864 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.865 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:374 3.es0 3:7924a6e6:::smithi10428818-20:head [read 0~2572288,checksum] snapc 0=[] ondisk+read+known_if_redirected e54)", 2020-03-09T00:25:55.865 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:55.865 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 51.943312665999997, 2020-03-09T00:25:55.865 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001, 2020-03-09T00:25:55.865 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.865 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:55.865 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.866 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.866 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.866 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 374 2020-03-09T00:25:55.866 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.866 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.866 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.867 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:55.867 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.867 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.867 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.867 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.867 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794143+0000", 2020-03-09T00:25:55.867 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.3215999999999998e-05 2020-03-09T00:25:55.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.868 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.868 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.868 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794216+0000", 2020-03-09T00:25:55.868 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.7970000000000003e-06 2020-03-09T00:25:55.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.869 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.869 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794223+0000", 2020-03-09T00:25:55.869 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2177e-05 2020-03-09T00:25:55.869 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.870 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.870 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794235+0000", 2020-03-09T00:25:55.870 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8439999999999997e-06 2020-03-09T00:25:55.870 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.870 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.870 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.871 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794240+0000", 2020-03-09T00:25:55.871 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7449999999999997e-05 2020-03-09T00:25:55.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.871 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.871 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794337+0000", 2020-03-09T00:25:55.871 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.0590000000000003e-06 2020-03-09T00:25:55.872 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.872 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.872 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:55.872 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.794345+0000", 2020-03-09T00:25:55.872 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0032359939999999998 2020-03-09T00:25:55.872 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.872 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.873 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.873 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797581+0000", 2020-03-09T00:25:55.873 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000114491 2020-03-09T00:25:55.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.797696+0000", 2020-03-09T00:25:55.874 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016180261000000001 2020-03-09T00:25:55.874 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.874 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.874 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.874 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:03.813876+0000", 2020-03-09T00:25:55.875 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.019733289000000001 2020-03-09T00:25:55.875 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.875 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.875 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.875 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.875 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.875 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:433 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~671744] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:55.876 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:55.876 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.936278739999999, 2020-03-09T00:25:55.876 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002, 2020-03-09T00:25:55.876 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.876 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.876 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.876 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.877 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.877 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 433 2020-03-09T00:25:55.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.877 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.877 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.877 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.877 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:55.878 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.878 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.878 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.878 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.878 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.801177+0000", 2020-03-09T00:25:55.878 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0030767469999999999 2020-03-09T00:25:55.878 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804253+0000", 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1990000000000006e-06 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.879 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804262+0000", 2020-03-09T00:25:55.880 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.9800999999999999e-05 2020-03-09T00:25:55.880 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.880 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.880 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.880 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804281+0000", 2020-03-09T00:25:55.881 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0343000000000001e-05 2020-03-09T00:25:55.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.881 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.882 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804292+0000", 2020-03-09T00:25:55.882 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000220564 2020-03-09T00:25:55.882 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.883 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.883 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804512+0000", 2020-03-09T00:25:55.883 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00012449100000000001 2020-03-09T00:25:55.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.883 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.883 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.883 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804637+0000", 2020-03-09T00:25:55.884 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00145191 2020-03-09T00:25:55.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.884 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.884 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806089+0000", 2020-03-09T00:25:55.884 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020643070999999999 2020-03-09T00:25:55.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.885 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.885 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.885 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826732+0000", 2020-03-09T00:25:55.885 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0081151190000000005 2020-03-09T00:25:55.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.886 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.886 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.886 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834847+0000", 2020-03-09T00:25:55.886 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.1645999999999996e-05 2020-03-09T00:25:55.886 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.886 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.887 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.887 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.834939+0000", 2020-03-09T00:25:55.887 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.033761891000000002 2020-03-09T00:25:55.887 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.887 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.887 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.887 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.888 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.888 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=66, reqid=client.4751.0:397, at_version=55'17, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:55.888 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:55.888 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.933091234999999, 2020-03-09T00:25:55.888 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001, 2020-03-09T00:25:55.888 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.888 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:55.889 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.889 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.889 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.889 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:55.889 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.890 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.890 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.890 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.890 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804364+0000", 2020-03-09T00:25:55.890 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000104311 2020-03-09T00:25:55.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.891 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.891 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804469+0000", 2020-03-09T00:25:55.891 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.5208999999999999e-05 2020-03-09T00:25:55.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.892 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.892 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804494+0000", 2020-03-09T00:25:55.892 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5976000000000001e-05 2020-03-09T00:25:55.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.893 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.893 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804510+0000", 2020-03-09T00:25:55.893 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0229999999999998e-06 2020-03-09T00:25:55.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.894 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.894 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.804516+0000", 2020-03-09T00:25:55.894 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0057666139999999998 2020-03-09T00:25:55.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.895 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.895 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810282+0000", 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.3182e-05 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810296+0000", 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016691906999999999 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.897 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.897 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.826987+0000", 2020-03-09T00:25:55.897 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8390000000000003e-05 2020-03-09T00:25:55.897 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.897 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.897 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827036+0000", 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.022671612000000001 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.898 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:434 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~770048] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:55.899 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:55.899 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.932349465999998, 2020-03-09T00:25:55.899 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003, 2020-03-09T00:25:55.899 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.899 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.899 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.899 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.900 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.900 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 434 2020-03-09T00:25:55.900 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.900 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.900 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.900 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.900 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:55.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.901 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.901 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.901 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.805106+0000", 2020-03-09T00:25:55.901 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0011664830000000001 2020-03-09T00:25:55.901 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806272+0000", 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.9829999999999999e-06 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.902 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806279+0000", 2020-03-09T00:25:55.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1960000000000001e-05 2020-03-09T00:25:55.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.903 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.903 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.903 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806291+0000", 2020-03-09T00:25:55.903 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.4999999999999999e-06 2020-03-09T00:25:55.903 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.904 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806297+0000", 2020-03-09T00:25:55.904 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0075845660000000001 2020-03-09T00:25:55.904 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.904 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.904 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.905 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.813882+0000", 2020-03-09T00:25:55.905 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013610299999999999 2020-03-09T00:25:55.905 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.905 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.905 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.906 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.814018+0000", 2020-03-09T00:25:55.906 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0047231749999999996 2020-03-09T00:25:55.906 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.906 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.906 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.818741+0000", 2020-03-09T00:25:55.907 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.01050245 2020-03-09T00:25:55.907 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.907 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.907 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.907 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829243+0000", 2020-03-09T00:25:55.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0085708439999999993 2020-03-09T00:25:55.908 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.908 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.908 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.908 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837814+0000", 2020-03-09T00:25:55.908 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4356000000000003e-05 2020-03-09T00:25:55.909 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.909 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.909 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.909 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.837898+0000", 2020-03-09T00:25:55.909 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.032792420000000003 2020-03-09T00:25:55.909 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.909 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.910 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.910 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.910 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.910 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:435 3.es0 3:73e16f4f:::smithi10428818-13:head [setxattr _header (57) in=7b] snapc 14=[14,13,11,10,f,c,5] ondisk+write+known_if_redirected e55)", 2020-03-09T00:25:55.910 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:55.910 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.931103692000001, 2020-03-09T00:25:55.910 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003, 2020-03-09T00:25:55.911 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.911 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.911 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.911 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.911 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.911 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 435 2020-03-09T00:25:55.911 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.912 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.912 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.912 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.912 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:55.912 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.912 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.913 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.913 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806352+0000", 2020-03-09T00:25:55.913 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.4781000000000001e-05 2020-03-09T00:25:55.913 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.913 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806427+0000", 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9479999999999996e-06 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806433+0000", 2020-03-09T00:25:55.914 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1838000000000001e-05 2020-03-09T00:25:55.915 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.915 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.915 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.915 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806444+0000", 2020-03-09T00:25:55.915 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7199999999999997e-06 2020-03-09T00:25:55.916 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.916 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.916 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.916 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806449+0000", 2020-03-09T00:25:55.916 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020656463999999999 2020-03-09T00:25:55.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.917 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.917 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.917 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827106+0000", 2020-03-09T00:25:55.917 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013190700000000001 2020-03-09T00:25:55.917 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.918 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827237+0000", 2020-03-09T00:25:55.918 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00078356200000000002 2020-03-09T00:25:55.918 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.918 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.918 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828021+0000", 2020-03-09T00:25:55.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.001223678 2020-03-09T00:25:55.919 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.919 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.919 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.919 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.829245+0000", 2020-03-09T00:25:55.919 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0087850050000000002 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838030+0000", 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.7771999999999998e-05 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.920 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.921 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838127+0000", 2020-03-09T00:25:55.921 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.031775675000000003 2020-03-09T00:25:55.921 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.921 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.921 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.921 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.921 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.922 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:436 3.es0 3:73e16f4f:::smithi10428818-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-03-09T00:25:55.922 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:55.922 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.930967197999998, 2020-03-09T00:25:55.922 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998, 2020-03-09T00:25:55.922 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.922 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "started", 2020-03-09T00:25:55.922 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.923 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.923 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.923 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 436 2020-03-09T00:25:55.923 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.923 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.923 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.923 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:55.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.924 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.924 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.924 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.924 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806488+0000", 2020-03-09T00:25:55.924 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.3732000000000001e-05 2020-03-09T00:25:55.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.925 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.925 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.926 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806532+0000", 2020-03-09T00:25:55.926 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5879999999999998e-06 2020-03-09T00:25:55.926 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.926 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.927 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.927 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806541+0000", 2020-03-09T00:25:55.927 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.2738000000000001e-05 2020-03-09T00:25:55.927 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.928 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.928 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806553+0000", 2020-03-09T00:25:55.928 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.605e-06 2020-03-09T00:25:55.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.929 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.929 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.806558+0000", 2020-03-09T00:25:55.929 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.021728470999999999 2020-03-09T00:25:55.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.930 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.930 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828286+0000", 2020-03-09T00:25:55.931 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000113791 2020-03-09T00:25:55.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.931 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting for rw locks", 2020-03-09T00:25:55.932 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.828400+0000", 2020-03-09T00:25:55.932 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.010563283 2020-03-09T00:25:55.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.933 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.933 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.838963+0000", 2020-03-09T00:25:55.933 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121599 2020-03-09T00:25:55.933 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.934 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.934 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.839085+0000", 2020-03-09T00:25:55.934 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0028128480000000002 2020-03-09T00:25:55.935 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.935 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.935 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.935 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.841898+0000", 2020-03-09T00:25:55.936 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.035409654999999998 2020-03-09T00:25:55.936 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.936 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.936 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.937 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.937 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.937 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "MOSDECSubOpWrite(3.7s1 55/54 ECSubWrite(tid=67, reqid=client.4751.0:398, at_version=55'18, trim_to=0'0, roll_forward_to=0'0))", 2020-03-09T00:25:55.937 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:55.938 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 50.927076423999999, 2020-03-09T00:25:55.938 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999, 2020-03-09T00:25:55.938 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.939 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "reached pg", 2020-03-09T00:25:55.939 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.939 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.939 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.940 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:55.940 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.940 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.941 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.941 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.941 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810379+0000", 2020-03-09T00:25:55.941 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000127076 2020-03-09T00:25:55.942 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.942 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.942 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.942 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810506+0000", 2020-03-09T00:25:55.943 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3247999999999998e-05 2020-03-09T00:25:55.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.943 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.944 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810529+0000", 2020-03-09T00:25:55.944 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.4992e-05 2020-03-09T00:25:55.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.945 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.945 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810544+0000", 2020-03-09T00:25:55.945 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.9669999999999998e-06 2020-03-09T00:25:55.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.946 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.946 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.810550+0000", 2020-03-09T00:25:55.946 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0098081420000000006 2020-03-09T00:25:55.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.947 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.947 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820358+0000", 2020-03-09T00:25:55.947 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5410999999999999e-05 2020-03-09T00:25:55.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.948 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.948 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.820374+0000", 2020-03-09T00:25:55.949 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0066733160000000003 2020-03-09T00:25:55.949 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.949 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.950 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827047+0000", 2020-03-09T00:25:55.950 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.4610999999999998e-05 2020-03-09T00:25:55.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.951 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.951 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:04.827082+0000", 2020-03-09T00:25:55.951 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.016702762999999999 2020-03-09T00:25:55.951 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.951 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.952 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.952 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.952 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.952 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:522 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:55.953 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:55.953 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.966705777000001, 2020-03-09T00:25:55.953 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018154887000000001, 2020-03-09T00:25:55.953 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.953 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.953 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.954 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.954 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.954 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 522 2020-03-09T00:25:55.954 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.954 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.955 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.955 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:55.955 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.955 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.955 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770750+0000", 2020-03-09T00:25:55.956 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.7240000000000002e-05 2020-03-09T00:25:55.956 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.956 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.956 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.956 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770797+0000", 2020-03-09T00:25:55.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.1880000000000002e-06 2020-03-09T00:25:55.957 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.957 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.957 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.957 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770805+0000", 2020-03-09T00:25:55.957 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.5597e-05 2020-03-09T00:25:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770821+0000", 2020-03-09T00:25:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.0109999999999997e-06 2020-03-09T00:25:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.959 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770827+0000", 2020-03-09T00:25:55.959 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00024047700000000001 2020-03-09T00:25:55.959 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.959 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.959 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.960 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771067+0000", 2020-03-09T00:25:55.960 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.0305e-05 2020-03-09T00:25:55.960 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.960 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.960 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "op must wait for map", 2020-03-09T00:25:55.961 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771078+0000", 2020-03-09T00:25:55.961 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.003975007 2020-03-09T00:25:55.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.962 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.962 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.962 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.775053+0000", 2020-03-09T00:25:55.962 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000152447 2020-03-09T00:25:55.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.963 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.963 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.775205+0000", 2020-03-09T00:25:55.963 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0010782299999999999 2020-03-09T00:25:55.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.776283+0000", 2020-03-09T00:25:55.964 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071855169999999998 2020-03-09T00:25:55.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.964 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.964 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783469+0000", 2020-03-09T00:25:55.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0053400169999999999 2020-03-09T00:25:55.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.965 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.965 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.788809+0000", 2020-03-09T00:25:55.965 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.5851000000000001e-05 2020-03-09T00:25:55.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.966 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.966 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.788905+0000", 2020-03-09T00:25:55.967 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018154887000000001 2020-03-09T00:25:55.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.968 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:523 3.es0 3:73e16f4f:::smithi10428818-13:head [rollback 1a] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:55.968 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:55.968 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.966584001000001, 2020-03-09T00:25:55.968 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018315774999999999, 2020-03-09T00:25:55.969 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.969 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.969 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.969 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.970 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.970 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 523 2020-03-09T00:25:55.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.971 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.971 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.971 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:55.971 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770871+0000", 2020-03-09T00:25:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.3540000000000002e-05 2020-03-09T00:25:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770895+0000", 2020-03-09T00:25:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 3.7960000000000002e-06 2020-03-09T00:25:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770899+0000", 2020-03-09T00:25:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 6.922e-06 2020-03-09T00:25:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770906+0000", 2020-03-09T00:25:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 2.9869999999999999e-06 2020-03-09T00:25:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770909+0000", 2020-03-09T00:25:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000235536 2020-03-09T00:25:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771144+0000", 2020-03-09T00:25:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.4800000000000001e-06 2020-03-09T00:25:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771153+0000", 2020-03-09T00:25:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0091831280000000005 2020-03-09T00:25:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.780336+0000", 2020-03-09T00:25:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00013816699999999999 2020-03-09T00:25:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.780474+0000", 2020-03-09T00:25:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00086282600000000002 2020-03-09T00:25:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.781337+0000", 2020-03-09T00:25:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.002215828 2020-03-09T00:25:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783553+0000", 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0055424929999999999 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.789095+0000", 2020-03-09T00:25:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 9.2071999999999995e-05 2020-03-09T00:25:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.789187+0000", 2020-03-09T00:25:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.018315774999999999 2020-03-09T00:25:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: "description": "osd_op(client.4751.0:524 3.es0 3:73e16f4f:::smithi10428818-13:head [append 0~0] snapc 1b=[1b,1a,15,13,c] ondisk+write+known_if_redirected e63)", 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: "initiated_at": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: "age": 42.966516675000001, 2020-03-09T00:25:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020037222, 2020-03-09T00:25:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: "type_data": { 2020-03-09T00:25:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: "flag_point": "commit sent; apply or cleanup", 2020-03-09T00:25:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: "client_info": { 2020-03-09T00:25:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: "client": "client.4751", 2020-03-09T00:25:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: "client_addr": "172.21.15.104:0/324735460", 2020-03-09T00:25:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: "tid": 524 2020-03-09T00:25:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: "events": [ 2020-03-09T00:25:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "initiated", 2020-03-09T00:25:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0 2020-03-09T00:25:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "throttled", 2020-03-09T00:25:55.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770939+0000", 2020-03-09T00:25:55.985 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.2867999999999997e-05 2020-03-09T00:25:55.985 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.985 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.985 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "header_read", 2020-03-09T00:25:55.985 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770982+0000", 2020-03-09T00:25:55.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 5.8850000000000004e-06 2020-03-09T00:25:55.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.986 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.986 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "all_read", 2020-03-09T00:25:55.986 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770988+0000", 2020-03-09T00:25:55.986 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 1.1586000000000001e-05 2020-03-09T00:25:55.986 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "dispatched", 2020-03-09T00:25:55.987 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.770999+0000", 2020-03-09T00:25:55.987 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 4.8590000000000004e-06 2020-03-09T00:25:55.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.987 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.987 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "queued_for_pg", 2020-03-09T00:25:55.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771004+0000", 2020-03-09T00:25:55.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000171402 2020-03-09T00:25:55.988 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.988 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.988 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.988 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771175+0000", 2020-03-09T00:25:55.988 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 7.2629999999999997e-06 2020-03-09T00:25:55.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.989 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "waiting_for_map not empty", 2020-03-09T00:25:55.989 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.771183+0000", 2020-03-09T00:25:55.989 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.011237143999999999 2020-03-09T00:25:55.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "reached_pg", 2020-03-09T00:25:55.990 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.782420+0000", 2020-03-09T00:25:55.990 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.000121636 2020-03-09T00:25:55.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.990 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "started", 2020-03-09T00:25:55.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.782541+0000", 2020-03-09T00:25:55.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00072058300000000001 2020-03-09T00:25:55.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.991 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_started", 2020-03-09T00:25:55.991 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783262+0000", 2020-03-09T00:25:55.991 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.00044685599999999998 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "sub_op_committed", 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.783709+0000", 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.0071821319999999999 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.992 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "commit_sent", 2020-03-09T00:25:55.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.790891+0000", 2020-03-09T00:25:55.993 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 8.5007999999999997e-05 2020-03-09T00:25:55.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2020-03-09T00:25:55.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2020-03-09T00:25:55.993 INFO:teuthology.orchestra.run.smithi148.stdout: "event": "done", 2020-03-09T00:25:55.993 INFO:teuthology.orchestra.run.smithi148.stdout: "time": "2020-03-09T00:25:12.790976+0000", 2020-03-09T00:25:55.994 INFO:teuthology.orchestra.run.smithi148.stdout: "duration": 0.020037222 2020-03-09T00:25:55.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.994 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2020-03-09T00:25:55.994 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2020-03-09T00:25:55.995 INFO:teuthology.orchestra.run.smithi148.stdout:} 2020-03-09T00:25:55.997 INFO:teuthology.orchestra.run.smithi148.stderr:nodeep-scrub is unset 2020-03-09T00:25:55.997 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2020-03-09T00:25:55.997 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-03-09T00:25:55.997 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:25:56.261 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:25:56.262 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:55.903860+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:25:59.274 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:25:59.545 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:25:59.546 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:55.903860+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:02.559 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:02.830 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:02.831 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:55.903860+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:05.844 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:06.115 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:06.115 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:55.903860+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:09.128 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:09.399 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:09.399 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:55.903860+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:12.412 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:12.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:12.682 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:55.903860+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:15.695 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:15.966 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:15.966 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:25:55.903860+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.078612+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:18.978 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:19.250 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:19.251 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":104,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:19.167233+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.981291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.320200+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.856110+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.954995+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.232010+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.899404+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:20.035026+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.800721+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.927465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:18.940015+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.252713+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:25:19.231940+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.166633+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:25:19.149062+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:22.264 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:22.531 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:22.531 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:25.544 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:25.815 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:25.816 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:28.829 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:29.119 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:29.119 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:32.132 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:32.400 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:32.400 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:35.412 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:35.683 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:35.684 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:38.697 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:38.968 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:38.968 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:41.980 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:42.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:42.248 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:45.260 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:45.530 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:45.530 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:48.542 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:48.812 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:48.813 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:51.825 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:52.094 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:52.094 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:55.107 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:55.379 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:55.380 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:26:58.392 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:26:58.662 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:26:58.662 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:01.674 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:01.951 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:01.951 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:04.964 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:05.234 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:05.235 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:08.247 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:08.517 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:08.518 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:11.530 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:11.799 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:11.800 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:14.812 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:15.081 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:15.082 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:18.094 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:18.365 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:18.365 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:26:21.172794+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.110494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.045897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.192069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.819218+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.560704+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.566663+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.220014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.116023+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.423612+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:19.579657+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:19.177937+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.062936+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:21.377 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:21.649 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:21.649 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:21.144740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.751195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:26:20.253995+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:26:20.895157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:24.661 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:24.932 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:24.933 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:27.945 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:28.214 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:28.215 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:31.227 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:31.495 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:31.496 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:34.508 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:34.776 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:34.777 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:37.789 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:38.060 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:38.060 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:41.073 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:41.342 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:41.343 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:44.356 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:44.625 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:44.625 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:47.638 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:47.909 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:47.909 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:50.921 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:51.192 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:51.192 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:54.204 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:54.472 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:54.473 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:27:57.485 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:27:57.755 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:27:57.755 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:00.768 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:01.036 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:01.036 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:04.049 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:04.319 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:04.320 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:07.332 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:07.600 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:07.600 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:10.613 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:10.880 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:10.881 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:13.893 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:14.161 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:14.161 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:17.173 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:17.442 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:17.443 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:27:22.149812+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.645097+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:19.700418+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.472427+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:20.455 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:20.728 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:20.729 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":110,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:19.821172+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.586349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.429857+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.193977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.074062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.199698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:27:21.305414+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.978842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:21.140778+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.815203+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.312871+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:19.415132+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:27:20.617670+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:23.741 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:24.010 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:24.011 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:27.024 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:27.294 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:27.294 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:30.306 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:30.576 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:30.577 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:33.589 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:33.861 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:33.861 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:36.874 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:37.143 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:37.143 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:40.156 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:40.425 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:40.426 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:43.439 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:43.709 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:43.709 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:46.722 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:46.992 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:46.992 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:50.005 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:50.274 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:50.275 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:53.287 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:53.557 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:53.558 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:56.571 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:28:56.841 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:28:56.841 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:28:59.853 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:00.122 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:00.123 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:03.135 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:03.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:03.403 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:06.415 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:06.685 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:06.686 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:09.699 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:09.969 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:09.970 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:12.981 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:13.251 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:13.251 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:16.264 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:16.533 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:16.534 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:19.547 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:19.817 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:19.818 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:28:22.835818+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.432366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.272746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.240075+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:21.344992+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.556313+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.978943+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.917146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.933993+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.230515+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.684724+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:28:19.759503+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:19.643877+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:20.778485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:22.831 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:23.102 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:23.103 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":116,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:22.800040+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:22.547793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:28:21.977815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:26.115 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:26.384 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:26.384 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:29.397 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:29.667 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:29.667 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:32.680 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:32.950 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:32.951 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:35.963 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:36.231 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:36.231 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:39.244 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:39.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:39.513 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:42.526 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:42.795 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:42.796 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:45.809 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:46.077 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:46.078 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:49.090 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:49.357 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:49.358 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:52.370 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:52.642 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:52.642 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:55.655 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:55.925 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:55.926 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:29:58.938 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:29:59.209 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:29:59.209 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:02.221 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:02.492 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:02.492 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:05.505 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:05.774 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:05.775 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:08.787 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:09.056 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:09.057 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:12.069 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:12.338 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:12.338 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:15.351 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:15.622 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:15.622 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:18.635 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:18.905 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:18.905 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:29:23.802048+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.472981+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.924185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.952545+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.252506+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:19.879395+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:20.742672+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:20.433956+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.645790+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:21.917 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:22.190 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:22.190 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:21.685891+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:23.536280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.328862+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.893313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:29:21.637915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.531601+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:22.736117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:29:21.834390+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:25.203 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:25.474 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:25.475 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:28.488 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:28.758 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:28.758 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:31.771 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:32.040 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:32.040 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:35.055 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:35.325 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:35.326 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:38.338 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:38.607 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:38.608 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:41.620 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:41.893 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:41.893 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:44.906 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:45.178 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:45.179 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:48.192 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:48.461 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:48.461 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:51.474 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:51.744 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:51.745 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:54.757 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:55.025 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:55.026 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:30:58.039 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:30:58.311 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:30:58.311 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:01.324 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:01.595 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:01.595 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:04.608 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:04.875 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:04.875 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:07.888 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:08.157 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:08.158 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:11.170 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:11.438 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:11.439 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:14.451 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:14.723 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:14.723 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:17.736 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:18.005 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:18.006 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:30:24.815296+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.401502+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:21.018 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:21.292 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:21.292 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":123,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:20.821759+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.338642+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.849810+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.148673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:22.424792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:22.760764+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.922660+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.228060+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.777403+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:30:20.815019+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:20.440108+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:21.673953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:24.305 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:24.574 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:24.575 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":126,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:23.834698+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:24.357997+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.344350+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:30:23.496154+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:27.588 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:27.861 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:27.862 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:30.874 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:31.143 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:31.144 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:34.156 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:34.426 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:34.426 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:37.439 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:37.708 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:37.708 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:40.721 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:40.989 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:40.990 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:44.002 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:44.270 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:44.271 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:47.283 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:47.555 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:47.555 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:50.568 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:50.838 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:50.839 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:53.851 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:54.121 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:54.122 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:31:57.134 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:31:57.404 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:31:57.405 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:00.417 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:00.687 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:00.688 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:03.700 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:03.977 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:03.978 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:06.991 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:07.262 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:07.263 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:10.275 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:10.544 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:10.545 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:13.557 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:13.826 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:13.827 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:16.839 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:17.108 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:17.109 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:20.121 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:20.390 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:20.390 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:31:24.838482+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.217456+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.854905+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:20.515298+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.532746+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:21.129662+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:21.381742+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:23.403 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:23.673 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:23.674 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":129,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:22.838420+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.530304+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.209687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.321524+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:31:22.726983+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:24.358312+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:23.769559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.834445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.146483+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:31:22.095336+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:26.686 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:26.958 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:26.959 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:29.972 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:30.244 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:30.245 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:33.258 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:33.528 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:33.529 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:36.542 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:36.812 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:36.812 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:39.826 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:40.093 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:40.094 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:43.107 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:43.376 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:43.376 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:46.389 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:46.655 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:46.656 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:49.668 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:49.941 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:49.942 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:52.954 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:53.222 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:53.222 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:56.236 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:56.506 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:56.506 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:32:59.519 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:32:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:32:59.787 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:02.800 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:03.075 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:03.075 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:06.086 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:06.354 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:06.355 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:09.367 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:09.636 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:09.637 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:12.649 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:12.917 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:12.918 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:15.930 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:16.202 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:16.203 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:19.212 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:19.481 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:19.481 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":132,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:32:25.851988+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.329612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.369197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:21.319788+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:22.494 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:22.763 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:22.764 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:22.432027+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.953942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.474242+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.985935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:32:22.870245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:24.962616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.819018+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.960534+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:23.078140+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.380211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:21.755261+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:22.947506+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:25.776 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:26.044 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:26.044 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":137,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:25.447263+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:32:25.415031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:29.057 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:29.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:29.328 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:32.343 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:32.613 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:32.613 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:35.625 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:35.898 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:35.898 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:38.911 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:39.180 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:39.181 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:42.194 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:42.460 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:42.460 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:45.473 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:45.743 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:45.744 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:48.756 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:49.079 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:33:49.078+0000 7ff83f7fe700 0 --2- 172.21.15.148:0/2932129009 >> v2:172.21.15.148:3300/0 conn(0x7ff8400bc640 0x7ff8400bca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T00:33:49.228 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:49.228 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:52.241 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:52.513 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:52.513 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:55.526 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:55.794 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:55.794 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:33:58.806 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:33:59.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:33:59.079 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:02.091 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:02.360 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:02.361 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:05.373 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:05.640 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:05.641 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:08.653 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:08.922 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:08.923 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:11.935 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:12.206 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:12.206 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:15.219 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:15.492 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:15.492 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:18.505 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:18.776 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:18.776 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:33:26.452594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.378502+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:21.788 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:22.058 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:22.059 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":139,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:21.617453+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.167360+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.562832+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:23.243537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.995459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.117443+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.191854+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:21.582295+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.788131+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:33:22.170925+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:22.767505+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:23.050860+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:25.071 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:25.342 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:25.342 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":142,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:24.630990+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.866156+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:24.536811+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:33:25.012881+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:28.355 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:28.623 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:28.624 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:31.636 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:31.905 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:31.905 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:34.918 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:35.188 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:35.189 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:38.201 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:38.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:38.473 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:41.486 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:41.755 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:41.755 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:44.768 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:45.037 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:45.038 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:48.050 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:48.321 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:48.322 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:51.335 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:51.604 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:51.604 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:54.616 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:54.884 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:54.884 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:34:57.897 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:34:58.168 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:34:58.168 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:01.181 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:01.452 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:01.452 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:04.466 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:04.736 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:04.736 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:07.749 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:08.019 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:08.019 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:11.032 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:11.302 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:11.302 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:14.315 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:14.588 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:14.588 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:17.600 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:17.873 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:17.874 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:20.886 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:21.153 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:21.154 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:34:27.644342+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.566330+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.104429+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.390541+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.401606+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:21.735715+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:22.947261+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.112524+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.745695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:24.166 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:24.438 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:24.439 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":147,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:24.219474+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.187436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.696294+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:34:23.570764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:25.106598+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:24.160004+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:23.804030+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:27.451 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:27.719 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:27.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":149,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:26.232125+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:34:26.807531+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:30.732 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:31.005 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:31.005 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:34.018 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:34.289 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:34.290 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:37.302 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:37.571 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:37.571 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:40.584 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:40.851 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:40.852 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:43.865 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:44.131 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:44.131 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:47.144 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:47.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:47.414 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:50.427 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:50.697 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:53.709 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:53.978 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:53.979 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:35:56.991 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:35:57.262 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:35:57.262 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:00.275 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:00.545 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:00.546 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:03.559 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:03.829 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:03.830 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:06.843 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:07.114 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:07.114 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:10.127 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:10.398 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:10.398 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:13.411 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:13.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:13.679 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:16.693 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:16.966 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:16.967 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:19.979 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:20.250 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:20.251 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:35:28.238099+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:21.608836+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:22.634380+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:23.263 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:23.532 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:23.533 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":152,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:22.887053+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.060673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.847706+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:24.521651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.307507+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.338695+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.582347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.635361+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.011589+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:35:23.271343+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:24.719227+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:23.962914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:26.545 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:26.815 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:26.816 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":155,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:25.903177+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:27.749037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:35:25.134795+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:29.827 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:30.099 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:30.099 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:33.111 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:33.383 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:33.383 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:36.397 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:36.666 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:36.666 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:39.679 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:39.948 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:39.948 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:42.961 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:43.231 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:43.232 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:46.244 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:46.514 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:46.514 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:49.527 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:49.797 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:49.798 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:52.811 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:53.081 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:53.081 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:56.094 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:56.365 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:56.365 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:36:59.378 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:36:59.652 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:36:59.653 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:02.665 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:02.935 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:02.936 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:05.948 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:06.223 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:06.224 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:09.242 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:09.510 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:09.511 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:12.524 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:12.795 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:12.796 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:15.808 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:16.079 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:16.079 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:19.091 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:19.359 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:19.360 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:22.372 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:22.643 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:22.643 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:36:28.913877+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:21.724457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.816415+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:24.687918+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.432589+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.657041+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.567822+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:22.862570+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:23.838108+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:36:23.493749+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:24.010956+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:25.655 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:25.924 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:25.950 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":161,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:25.898734+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:26.046198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.676236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.547984+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:25.646484+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:28.955 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:29.223 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:29.224 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":162,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:26.902781+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:36:28.456107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:32.236 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:32.507 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:32.507 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:35.520 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:35.790 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:35.790 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:38.803 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:39.071 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:39.072 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:42.084 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:42.353 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:42.354 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:45.365 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:45.635 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:45.635 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:48.648 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:48.916 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:48.916 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:51.929 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:52.198 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:52.198 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:55.210 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:55.480 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:55.480 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:37:58.492 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:37:58.763 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:37:58.763 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:01.776 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:02.045 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:02.046 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:05.057 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:05.325 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:05.326 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:08.340 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:08.611 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:08.612 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:11.624 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:11.893 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:11.894 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:14.906 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:15.177 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:15.178 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:18.189 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:18.461 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:18.461 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:21.474 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:21.744 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:21.745 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:37:29.913738+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.123088+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.563592+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:22.959756+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:23.991868+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:23.525648+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.071109+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:24.757 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:25.027 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:25.028 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":166,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:24.890688+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.081940+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.601794+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.300235+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:37:25.366611+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.956654+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:24.677517+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:25.372557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:26.545402+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:28.040 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:28.309 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:28.310 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":169,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:27.904766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:37:29.429205+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:31.322 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:31.595 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:31.595 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:34.604 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:34.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:34.875 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:37.887 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:38.160 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:38.160 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:41.173 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:41.442 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:41.442 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:44.455 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:44.726 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:44.727 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:47.739 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:48.008 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:48.009 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:51.021 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:51.289 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:51.290 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:54.302 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:54.570 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:54.571 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:38:57.583 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:38:57.854 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:38:57.854 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:00.867 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:01.138 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:01.139 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:04.152 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:04.422 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:04.423 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:07.435 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:07.703 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:07.704 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:10.717 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:10.987 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:10.988 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:14.000 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:14.269 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:14.270 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:17.282 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:17.552 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:17.553 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:20.565 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:20.834 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:20.834 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":170,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:38:29.911483+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:22.145244+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:23.847 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:24.116 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:24.116 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":171,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:23.186922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.216405+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:26.118265+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.816852+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:25.130897+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.636958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.173091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:23.778158+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.777942+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:38:23.939485+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.728091+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:24.864004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:27.129 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:27.398 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:27.398 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":175,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:27.202673+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:29.496448+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:27.058016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:38:26.814188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:30.411 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:30.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:30.682 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:33.694 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:33.963 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:33.964 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:36.977 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:37.246 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:37.246 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:40.259 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:40.528 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:40.528 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:43.540 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:43.808 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:43.808 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:46.820 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:47.090 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:47.090 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:50.103 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:50.373 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:50.373 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:53.385 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:53.652 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:53.653 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:56.666 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:39:56.936 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:39:56.936 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:39:59.950 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:00.219 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:00.219 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:03.231 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:03.501 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:03.502 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:06.514 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:06.786 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:06.786 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:09.799 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:10.069 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:10.070 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:13.082 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:13.351 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:13.351 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:16.365 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:16.634 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:16.634 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:19.646 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:19.917 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:19.917 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:22.930 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:23.197 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:23.198 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":177,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:39:30.211964+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:23.135650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:25.433210+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.109737+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.903321+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:24.586151+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:24.872596+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:26.210 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:26.478 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:26.479 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":180,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:26.256926+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.522186+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.249116+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:39:27.020115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.055943+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.122017+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:26.188320+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:27.001678+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:29.491 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:29.762 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:29.762 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":182,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:28.907070+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:29.672263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:39:28.024358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:32.774 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:33.043 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:33.043 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:36.056 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:36.324 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:36.325 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:39.337 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:39.607 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:39.608 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:42.620 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:42.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:42.889 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:45.901 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:46.171 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:46.172 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:49.184 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:49.452 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:49.453 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:52.465 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:52.736 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:52.736 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:55.750 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:56.021 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:56.021 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:40:59.034 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:40:59.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:40:59.304 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:02.317 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:02.587 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:02.587 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:05.600 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:05.869 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:05.870 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:08.882 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:09.151 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:09.151 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:12.164 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:12.433 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:12.434 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:15.447 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:15.717 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:15.717 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:18.730 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:19.001 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:19.002 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:22.014 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:22.284 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:22.284 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":183,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:40:29.909090+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:23.651573+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:25.297 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:25.567 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:25.567 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":184,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:23.946337+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.219532+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.236997+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.127347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:26.071777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:24.175335+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.897293+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:24.815357+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:25.625234+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:28.580 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:28.850 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:28.850 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":187,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:27.960980+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.834953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:29.057212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:28.277993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:40:27.936526+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.107867+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:40:27.953929+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:31.862 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:32.134 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:35.147 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:35.417 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:35.418 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:38.434 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:38.707 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:38.708 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:41.720 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:41.991 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:41.991 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:45.004 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:45.273 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:45.273 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:48.286 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:48.558 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:48.559 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:51.571 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:51.841 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:51.842 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:54.854 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:55.123 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:55.123 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:41:58.135 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:41:58.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:41:58.409 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:01.428 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:01.700 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:01.701 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:04.714 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:04.981 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:04.982 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:07.995 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:08.273 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:08.273 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:11.285 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:11.554 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:14.566 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:14.836 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:14.836 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:17.848 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:18.122 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:18.122 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:21.134 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:21.405 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:21.406 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":189,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:41:30.974431+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:23.747382+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:24.418 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:24.686 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:24.687 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":190,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:23.872356+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.400473+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.151071+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.025018+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:25.038384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.771926+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:25.229397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:26.564856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:27.699 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:27.970 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:27.980 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":193,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:27.925963+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.616124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:30.003016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.286732+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:41:28.662427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:27.285976+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.071800+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:41:28.613399+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:30.984 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:31.253 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:34.265 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:34.536 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:34.537 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:37.549 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:37.818 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:37.819 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:40.831 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:41.098 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:41.099 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:44.111 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:44.377 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:44.378 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:47.391 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:47.661 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:47.662 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:50.680 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:50.950 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:50.950 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:53.963 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:54.231 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:54.232 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:42:57.244 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:42:57.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:42:57.512 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:00.525 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:00.795 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:00.795 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:03.808 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:04.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:04.078 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:07.091 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:07.360 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:07.361 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:10.373 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:10.642 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:10.643 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:13.655 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:13.923 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:13.923 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:16.936 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:17.207 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:17.208 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:20.221 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:20.492 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:20.493 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:23.505 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:23.775 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:23.775 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":196,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:42:30.937724+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:23.820947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:25.546711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:26.788 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:27.062 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:27.063 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":198,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:26.832045+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.210280+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.582694+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.965250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.325473+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:27.210855+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:28.769500+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.796741+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:25.993577+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:26.766214+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:30.076 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:30.347 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:30.347 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":201,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:29.941612+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.781222+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:30.254024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:42:29.465937+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:42:29.606375+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:33.359 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:33.626 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:33.627 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:36.639 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:36.909 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:36.910 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:39.922 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:40.192 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:40.192 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:43.205 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:43.475 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:43.476 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:46.488 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:46.759 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:46.759 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:49.772 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:50.043 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:50.043 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:53.055 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:53.323 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:53.324 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:56.336 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:56.606 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:56.607 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:43:59.619 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:43:59.887 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:43:59.888 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:02.900 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:03.170 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:03.171 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:06.182 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:06.453 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:06.454 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:09.467 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:09.738 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:09.739 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:12.751 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:13.020 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:13.020 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:16.033 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:16.300 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:16.301 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:19.314 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:19.584 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:19.584 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:22.597 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:22.867 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:22.868 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":203,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:43:31.953644+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:24.618538+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:25.880 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:26.150 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:26.151 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":204,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:25.489799+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.679621+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.418665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:28.117813+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.230933+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:26.372941+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.796709+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:26.939089+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:27.473106+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:29.163 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:29.432 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:29.433 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":206,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:28.947111+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.274251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.591022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:43:30.235794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:29.246607+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:30.339233+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:32.445 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:32.715 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:32.715 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":209,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:31.962366+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:43:31.766815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:35.728 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:36.001 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:36.001 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:39.014 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:39.283 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:39.284 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:42.296 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:42.568 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:42.569 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:45.581 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:45.852 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:45.853 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:48.866 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:49.137 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:49.137 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:52.150 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:52.420 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:52.421 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:55.433 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:55.702 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:55.703 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:44:58.715 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:44:58.985 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:44:58.985 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:01.998 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:02.266 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:02.267 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:05.278 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:05.549 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:05.549 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:08.562 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:08.831 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:08.831 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:11.843 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:12.111 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:12.112 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:15.124 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:15.394 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:15.395 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:18.407 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:18.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:18.680 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:21.692 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:21.962 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:21.962 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:24.975 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:25.245 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:25.246 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":210,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:44:32.968659+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:25.438490+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.843867+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.304314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:27.370437+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:27.389869+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:28.258 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:28.530 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:28.530 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":212,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:28.301798+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.664530+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:44:30.397108+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.546250+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.144854+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:29.921815+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.620083+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:30.478483+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:28.270556+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:31.543 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:31.815 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:31.815 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":214,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:31.318173+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:32.462599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:44:31.252485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:34.827 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:35.095 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:35.096 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:38.108 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:38.378 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:38.378 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:41.391 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:41.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:41.660 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:44.673 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:44.943 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:44.944 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:47.957 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:48.225 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:48.226 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:51.238 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:51.506 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:51.507 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:54.519 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:54.789 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:54.790 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:45:57.802 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:45:58.074 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:45:58.075 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:01.087 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:01.356 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:01.357 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:04.370 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:04.638 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:04.639 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:07.651 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:07.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:07.921 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:10.934 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:11.204 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:11.205 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:14.218 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:14.486 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:14.486 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:17.498 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:17.770 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:17.770 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:20.783 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:21.052 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:21.053 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:24.065 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:24.332 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:24.333 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":216,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:45:33.327212+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:25.618973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:27.346 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:27.615 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:27.616 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":217,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:25.968896+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.057509+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.589504+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:29.028325+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.077115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.170935+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.728474+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:27.549318+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:28.495191+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:30.628 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:30.896 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:30.896 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":219,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:29.983158+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:32.816174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:31.841859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.471194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:45:30.514157+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.837430+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:45:30.496772+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:33.909 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:34.178 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:34.179 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:37.191 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:37.460 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:37.460 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:40.473 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:40.744 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:43.756 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:44.027 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:44.027 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:47.040 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:47.310 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:47.311 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:50.323 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:50.593 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:50.593 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:53.606 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:53.878 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:53.878 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:46:56.890 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:46:57.160 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:46:57.161 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:00.173 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:00.440 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:00.441 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:03.453 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:03.722 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:03.723 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:06.734 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:07.004 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:07.005 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:10.018 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:10.292 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:10.292 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:13.305 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:13.574 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:13.574 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:16.586 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:16.857 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:16.857 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:19.870 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:20.139 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:20.140 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:23.152 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:23.422 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:23.423 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:26.435 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:26.705 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:26.706 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":221,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:46:32.994922+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:25.621167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.374777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.364954+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:29.718 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:29.986 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:29.986 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":223,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:29.045274+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.120592+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.643311+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:30.665862+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.342200+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.172954+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:28.679900+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.901895+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.516457+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:46:28.415245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:30.541497+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:29.449916+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:32.998 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:33.270 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:33.271 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":227,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:33.067171+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:46:32.960181+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:36.284 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:36.553 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:36.553 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:39.566 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:39.835 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:39.836 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:42.849 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:43.119 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:43.119 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:46.131 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:46.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:46.402 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:49.415 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:49.683 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:49.684 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:52.696 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:52.968 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:52.968 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:55.978 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:56.252 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:56.252 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:47:59.265 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:47:59.537 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:47:59.537 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:02.550 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:02.874 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:48:02.872+0000 7f6179bab700 0 --2- 172.21.15.148:0/3923812223 >> v2:172.21.15.50:3300/0 conn(0x7f6160004db0 0x7f6174142c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T00:48:03.022 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:03.022 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:06.034 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:06.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:06.303 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:09.316 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:09.587 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:09.587 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:12.600 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:12.868 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:12.869 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:15.881 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:16.153 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:16.153 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:19.166 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:19.438 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:19.438 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:22.450 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:22.719 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:22.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:25.732 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:26.003 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":228,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:47:34.071857+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:26.272465+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.436256+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:29.015 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:29.285 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:29.286 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":230,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:29.058386+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.772616+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:28.654246+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:31.027200+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.222361+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.755917+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.189717+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.532336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:47:29.193000+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:30.599213+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:29.618974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:32.299 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:32.570 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:32.570 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":233,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:32.071164+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:33.830781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:32.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:47:31.729643+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:35.583 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:35.848 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:35.849 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:38.861 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:39.129 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:39.130 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:42.142 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:42.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:42.413 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:45.425 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:45.693 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:45.693 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:48.705 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:48.978 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:48.979 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:51.992 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:52.261 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:52.261 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:55.274 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:55.548 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:55.549 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:48:58.560 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:48:58.833 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:48:58.834 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:01.846 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:02.114 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:02.115 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:05.127 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:05.396 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:05.396 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:08.409 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:08.677 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:08.677 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:11.691 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:11.960 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:11.960 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:14.973 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:15.244 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:15.244 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:18.256 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:18.528 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:18.529 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:21.540 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:21.810 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:21.810 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:24.823 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:25.095 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:25.095 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":235,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:48:34.079489+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:27.060119+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:28.108 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:28.378 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:28.378 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":236,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:28.149230+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.535182+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.996923+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:28.486542+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:29.668555+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:30.140591+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.507611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:31.390 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:31.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:31.661 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":239,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:31.163801+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:33.322824+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.507926+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:48:31.994944+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:31.221122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.592433+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:32.491494+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:30.954965+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:34.673 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:34.944 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:34.945 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":242,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:34.179887+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:48:34.075641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:37.958 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:38.229 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:38.230 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:41.243 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:41.513 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:41.513 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:44.527 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:44.793 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:44.794 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:47.806 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:48.078 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:51.090 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:51.359 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:51.359 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:54.372 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:54.641 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:54.642 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:49:57.653 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:49:57.925 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:49:57.926 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:00.938 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:01.210 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:01.210 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:04.222 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:04.495 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:04.495 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:07.509 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:07.778 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:07.778 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:10.790 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:11.061 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:11.062 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:14.074 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:14.345 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:14.345 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:17.357 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:17.626 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:17.627 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:20.639 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:20.909 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:20.910 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:23.922 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:24.191 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:24.192 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:27.204 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:27.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:27.474 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":243,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:49:35.183313+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:27.997054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:28.781094+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:30.487 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:30.758 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:30.759 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":244,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:29.082227+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.837187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.097711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.824382+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.414311+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.576869+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:29.944533+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:31.107954+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:31.392850+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:30.520934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:33.771 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:34.039 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:34.040 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":247,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:33.098171+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:34.381000+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.632045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:49:32.767308+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:49:33.107498+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:37.052 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:37.323 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:37.323 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:40.335 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:40.603 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:40.604 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:43.617 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:43.888 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:43.888 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:46.900 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:47.170 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:47.171 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:50.183 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:50.453 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:50.453 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:53.466 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:53.738 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:53.738 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:50:56.750 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:50:57.020 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:50:57.020 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:00.033 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:00.302 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:00.303 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:03.316 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:03.585 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:03.586 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:06.598 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:06.870 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:06.871 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:09.884 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:10.154 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:10.154 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:13.166 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:13.435 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:13.436 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:16.449 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:16.719 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:16.719 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:19.732 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:20.000 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:20.001 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:23.013 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:23.280 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:23.280 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:26.293 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:26.563 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:26.563 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":249,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:50:35.108772+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.784873+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:28.901622+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:29.575 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:29.844 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:29.844 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":250,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:29.094899+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.351700+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.437796+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.654520+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:30.119249+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:31.157548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.529574+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:31.319451+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:32.857 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:33.128 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":252,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:32.106513+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.395937+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:34.381210+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.185903+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:50:33.599815+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:32.777524+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:50:33.383550+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:36.142 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:36.411 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:36.412 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:39.424 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:39.695 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:39.696 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:42.708 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:42.976 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:42.977 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:45.989 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:46.258 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:46.259 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:49.271 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:49.542 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:49.543 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:52.555 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:52.823 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:52.824 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:55.836 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:56.107 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:56.108 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:51:59.120 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:51:59.388 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:51:59.388 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:02.400 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:02.667 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:02.668 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:05.680 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:05.949 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:05.950 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:08.962 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:09.232 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:09.232 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:12.244 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:12.519 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:12.520 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:15.533 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:15.805 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:15.806 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:18.818 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:19.088 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:19.088 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:22.101 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:22.370 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:22.371 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:25.383 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:25.650 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:25.651 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:28.663 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:28.933 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:28.934 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":254,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:51:35.113768+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.899080+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.265441+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:28.919984+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:30.440211+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:31.946 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:32.218 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:32.218 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":257,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:31.719456+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.055370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:33.621109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.349861+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.016457+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.611569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:33.506149+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:51:32.061234+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:31.793540+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:32.033480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:35.230 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:35.498 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:35.499 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":260,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:35.045005+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.597442+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:51:34.996657+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:38.512 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:38.781 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:38.782 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:41.793 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:42.066 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:42.067 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:45.080 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:45.352 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:45.352 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:48.365 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:48.634 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:48.635 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:51.648 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:51.917 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:51.918 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:54.930 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:55.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:55.199 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:52:58.211 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:52:58.483 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:52:58.484 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:01.497 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:01.767 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:01.767 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:04.779 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:05.050 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:08.063 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:08.336 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:08.336 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:11.348 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:11.617 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:11.618 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:14.630 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:14.903 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:14.904 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:17.917 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:18.191 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:18.192 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:21.204 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:21.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:21.474 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:24.487 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:24.754 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:24.755 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:27.768 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:28.036 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:28.037 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":261,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:52:36.048642+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.921545+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:29.678323+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:31.050 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:31.317 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:31.317 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":263,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:30.878233+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:31.431698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.803515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.572445+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:30.504369+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:32.278493+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.503404+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:32.163750+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:34.331 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:34.599 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:34.599 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":266,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:34.049269+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.624004+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:35.782214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.027085+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:52:34.608050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:34.301019+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:52:33.757744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:37.612 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:37.881 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:40.894 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:41.164 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:41.165 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:44.178 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:44.450 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:44.451 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:47.463 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:47.733 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:47.734 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:50.746 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:51.013 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:51.013 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:54.025 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:54.293 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:54.294 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:53:57.307 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:53:57.578 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:53:57.579 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:00.591 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:00.861 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:00.862 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:03.874 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:04.144 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:04.145 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:07.157 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:07.432 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:07.433 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:10.439 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:10.710 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:10.710 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:13.723 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:13.993 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:13.993 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:17.006 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:17.328 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T00:54:17.325+0000 7f8aaf7fe700 0 --2- 172.21.15.148:0/1204546611 >> v2:172.21.15.50:3300/0 conn(0x7f8ab0141ef0 0x7f8ab0148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T00:54:17.480 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:17.481 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:20.493 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:20.763 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:20.763 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:23.775 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:24.044 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:24.044 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:27.057 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:27.325 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:27.326 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:30.339 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:30.609 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:30.609 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":269,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:53:37.066656+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:30.218770+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.714379+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.644160+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:29.824640+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:31.377631+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:33.622 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:33.893 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:33.893 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":272,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:33.147366+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:35.965661+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:36.122244+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.986624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:35.128264+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.442078+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.140850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:34.610014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:53:32.367875+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:33.416547+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:53:32.231445+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:36.906 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:37.179 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:37.179 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:40.192 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:40.464 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:40.464 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:43.477 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:43.745 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:43.746 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:46.758 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:47.027 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:47.027 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:50.040 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:50.309 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:50.310 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:53.322 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:53.590 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:53.591 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:56.603 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:54:56.875 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:54:56.876 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:54:59.888 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:00.157 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:00.158 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:03.170 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:03.440 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:03.441 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:06.454 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:06.726 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:06.726 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:09.739 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:10.009 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:10.010 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:13.022 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:13.291 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:13.291 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:16.303 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:16.574 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:19.587 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:19.854 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:19.855 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:22.868 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:23.141 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:23.142 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:26.154 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:26.421 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:26.421 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:29.433 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:29.703 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:29.703 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":276,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:54:37.167766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.033226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:31.749332+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:30.725197+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.069337+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:32.716 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:32.988 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:32.988 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":278,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:32.756735+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.546327+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.272906+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:32.944285+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:34.740571+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:33.238199+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.440024+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:33.220665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:36.002 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:36.271 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:36.271 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":281,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:36.068252+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.812936+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:36.749216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:54:35.862578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:54:35.299942+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:39.283 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:39.552 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:39.553 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:42.565 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:42.831 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:42.831 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:45.843 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:46.113 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:46.113 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:49.126 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:49.394 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:49.394 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:52.407 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:52.676 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:52.677 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:55.690 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:55.958 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:55.959 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:55:58.971 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:55:59.242 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:55:59.242 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:02.255 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:02.524 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:02.525 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:05.537 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:05.808 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:05.808 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:08.820 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:09.089 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:09.090 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:12.102 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:12.370 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:12.371 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:15.384 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:15.653 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:15.653 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:18.666 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:18.937 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:18.938 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:21.949 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:22.219 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:22.220 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:25.233 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:25.504 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:25.505 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:28.517 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:28.786 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:28.786 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":283,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:55:38.078546+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:30.945890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:31.799 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:32.072 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:32.072 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":284,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:31.682946+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.014509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.618357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.527572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.880575+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:32.403510+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:34.044068+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:34.093821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:33.714977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:35.086 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:35.354 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:35.355 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":287,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:35.088397+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.887837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:37.158935+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:36.734638+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:55:36.238616+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.423349+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:55:35.482924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:38.368 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:38.639 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:38.639 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:41.651 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:41.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:41.921 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:44.933 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:45.202 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:45.202 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:48.214 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:48.486 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:48.487 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:51.498 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:51.768 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:51.768 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:54.781 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:55.051 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:56:58.064 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:56:58.336 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:56:58.336 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:01.348 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:01.621 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:01.621 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:04.633 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:04.903 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:04.904 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:07.915 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:08.187 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:08.188 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:11.200 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:11.472 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:11.472 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:14.484 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:14.763 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:14.763 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:17.775 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:18.044 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:18.045 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:21.057 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:21.325 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:21.325 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:24.339 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:24.609 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:24.610 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:27.621 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:27.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:27.890 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:30.902 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:31.169 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:31.169 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":290,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:56:38.103552+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:32.106913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.391219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:31.631670+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:34.182 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:34.450 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:34.450 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":292,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:34.091576+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.849164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:35.701645+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.227855+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.805063+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:33.432927+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:34.112876+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.463958+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:34.653183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:37.463 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:37.730 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:37.730 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":295,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:37.105208+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.224953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:37.396834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:56:37.201697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:56:36.386609+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:40.743 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:41.011 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:41.012 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:44.023 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:44.297 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:47.310 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:47.580 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:47.580 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:50.593 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:50.863 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:50.863 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:53.876 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:54.145 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:54.145 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:57:57.157 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:57:57.426 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:57:57.426 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:00.439 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:00.709 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:00.709 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:03.722 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:03.992 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:03.992 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:07.005 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:07.277 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:07.278 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:10.290 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:10.560 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:10.560 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:13.572 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:13.841 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:13.842 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:16.854 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:17.125 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:17.126 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:20.138 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:20.411 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:20.412 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:23.424 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:23.694 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:23.695 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:26.707 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:26.976 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:26.977 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:29.989 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:30.262 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:30.263 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":297,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:57:39.116472+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.967572+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:33.271 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:33.541 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:33.541 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":298,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:33.101606+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:33.499384+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.148290+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:32.442979+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.359104+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:34.126134+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.226366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:34.807957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:36.555 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:36.824 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:36.825 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":301,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:36.114444+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.075999+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.071767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:57:37.529812+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:36.480766+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:35.529570+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:37.118065+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:39.836 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:40.105 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:40.106 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":304,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:39.127006+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:57:38.346372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:43.119 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:43.387 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:43.388 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:46.401 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:46.674 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:46.674 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:49.687 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:49.958 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:49.959 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:52.971 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:53.239 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:53.240 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:56.252 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:56.522 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:56.522 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:58:59.535 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:58:59.804 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:58:59.804 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:02.817 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:03.086 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:03.087 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:06.099 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:06.372 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:06.372 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:09.385 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:09.653 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:09.654 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:12.666 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:12.936 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:12.937 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:15.949 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:16.227 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:16.228 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:19.240 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:19.511 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:19.512 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:22.524 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:25.808 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:26.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:26.079 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:29.090 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:29.360 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:29.361 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:32.374 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:32.645 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:32.645 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":305,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:58:40.134302+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.016057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.698478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:33.316955+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:34.898392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:35.658 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:35.924 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:35.925 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":307,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:35.118460+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:38.783861+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.325047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.733419+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.252329+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.394391+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:37.795404+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.307322+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:36.028037+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:35.281941+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:38.938 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:39.211 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:39.212 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":311,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:39.136844+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:58:39.270796+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:58:38.484137+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:42.224 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:42.491 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:42.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:45.504 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:45.775 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:48.787 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:49.059 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:49.059 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:52.071 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:52.340 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:52.340 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:55.353 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:55.622 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:55.622 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T00:59:58.635 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T00:59:58.905 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T00:59:58.906 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:01.918 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:02.188 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:02.189 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:05.201 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:05.469 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:05.470 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:08.482 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:08.752 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:08.752 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:11.765 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:12.035 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:12.036 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:15.048 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:15.317 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:15.318 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:18.331 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:18.600 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:18.600 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:21.613 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:21.883 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:21.884 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:24.896 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:25.164 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:25.164 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:28.176 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:28.445 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:28.446 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:31.459 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:31.729 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:31.730 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":313,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T00:59:41.145309+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.843495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:33.464353+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:34.742 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:35.008 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:35.009 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":314,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:34.122178+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:34.506708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.862041+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.242967+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.119436+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.903894+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.531336+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:35.056189+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:36.062301+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:35.286188+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:38.021 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:38.291 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:38.381 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":317,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:38.134534+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:38.805191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:40.201291+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:59:37.391422+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T00:59:39.309279+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:41.384 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:41.654 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:41.654 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:44.667 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:44.936 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:44.937 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:47.949 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:48.220 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:48.221 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:51.233 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:51.503 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:51.504 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:54.516 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:54.785 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:54.786 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:00:57.798 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:00:58.066 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:00:58.066 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:01.079 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:01.349 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:01.350 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:04.361 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:04.681 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T01:01:04.680+0000 7fa9d6ffd700 0 --2- 172.21.15.148:0/674798991 >> v2:172.21.15.148:3300/0 conn(0x7fa9d80ba300 0x7fa9d814c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T01:01:04.832 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:04.832 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:07.844 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:08.113 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:08.114 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:11.126 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:11.395 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:11.396 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:14.409 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:14.678 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:14.679 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:17.691 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:17.961 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:17.961 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:20.974 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:21.244 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:21.244 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:24.256 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:24.525 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:24.526 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:27.538 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:27.807 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:27.808 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:30.820 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:31.090 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:31.091 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":320,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:00:41.148085+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:33.733201+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:34.103 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:34.374 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:34.374 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":321,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:34.132462+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:34.007650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.308325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.418282+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:35.779392+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.406449+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:35.339239+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:37.387 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:37.657 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:37.658 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":324,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:37.146387+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:39.156550+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.214065+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.979877+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:38.018095+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:37.915917+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:36.244782+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:40.670 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:40.940 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:40.940 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":327,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:40.158358+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:00:41.046381+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:00:39.362747+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:43.953 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:44.224 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:44.225 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:47.238 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:47.507 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:50.519 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:50.789 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:50.789 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:53.802 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:54.072 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:54.072 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:01:57.084 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:01:57.355 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:01:57.355 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:00.367 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:00.636 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:00.636 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:03.649 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:03.919 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:03.919 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:06.931 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:07.200 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:07.200 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:10.214 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:10.483 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:10.484 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:13.496 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:13.762 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:13.762 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:16.775 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:17.051 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:17.052 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:20.064 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:20.335 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:20.335 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:23.347 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:23.616 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:23.616 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:26.628 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:26.900 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:26.900 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:29.912 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:30.181 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:30.182 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:33.194 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:33.464 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":329,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:01:42.167224+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:34.217948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:33.808592+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:36.477 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:36.746 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:36.747 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":330,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:35.144068+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.070992+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.809059+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.584955+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.579221+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:37.199142+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:35.904430+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.724311+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:36.013726+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:39.760 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:40.029 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:40.030 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":333,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:39.162400+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.995812+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:42.007289+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:39.067612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:01:40.359849+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:01:38.976980+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:43.041 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:43.312 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:43.312 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:46.325 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:46.594 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:46.594 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:49.607 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:49.875 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:49.875 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:52.888 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:53.157 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:56.170 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:56.441 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:56.441 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:02:59.454 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:02:59.728 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:02:59.729 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:02.741 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:03.011 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:03.012 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:06.024 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:06.294 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:06.295 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:09.307 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:09.578 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:09.578 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:12.591 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:12.860 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:12.860 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:15.872 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:16.143 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:16.144 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:19.157 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:19.427 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:19.428 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:22.440 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:22.708 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:22.709 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:25.723 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:25.994 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:25.994 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:29.007 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:29.280 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:29.280 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:32.293 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:32.562 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:32.563 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":336,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:02:42.174852+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.218860+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:35.575 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:35.843 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:35.844 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":337,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:35.263846+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.248373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.590241+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:34.774166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:37.244498+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:36.539394+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.888009+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:36.195453+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:38.856 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:39.127 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:39.128 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":340,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:38.273707+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:40.493184+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:42.015343+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.165905+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:02:41.267269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.507069+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:39.270777+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:02:38.604566+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:42.140 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:42.416 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:42.416 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:45.430 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:45.700 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:45.701 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:48.713 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:48.985 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:48.986 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:51.998 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:52.267 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:52.268 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:55.280 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:55.550 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:55.551 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:03:58.563 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:03:58.833 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:03:58.833 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:01.846 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:02.116 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:02.117 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:05.130 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:05.403 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:05.403 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:08.416 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:08.684 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:08.685 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:11.697 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:11.967 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:11.967 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:14.979 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:15.249 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:15.249 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:18.261 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:18.531 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:18.532 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:21.545 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:21.812 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:21.813 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:24.826 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:25.092 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:25.093 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:28.106 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:28.376 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:28.376 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:31.389 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:31.659 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:31.660 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:34.672 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:34.939 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:34.940 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":343,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:03:42.287491+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.112077+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:35.756290+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:36.584953+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.217642+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:37.952 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:38.221 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:38.221 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":345,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:37.285285+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.369151+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:36.543744+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.268790+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.218560+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.365452+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:39.424379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:38.247009+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:37.891411+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:41.234 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:41.507 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:41.507 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":349,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:41.301650+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:41.351815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:03:42.039382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:03:42.243383+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:44.520 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:44.788 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:44.788 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:47.801 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:48.069 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:48.070 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:51.083 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:51.352 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:51.353 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:54.365 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:54.636 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:54.637 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:04:57.649 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:04:57.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:04:57.920 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:00.933 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:01.205 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:01.205 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:04.217 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:04.486 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:04.487 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:07.499 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:07.769 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:07.770 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:10.783 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:11.053 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:11.054 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:14.066 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:14.337 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:14.337 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:17.349 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:17.619 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:17.620 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:20.632 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:20.904 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:20.904 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:23.917 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:24.186 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:24.187 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:27.199 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:27.469 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:27.469 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:30.482 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:30.750 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:30.750 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:33.762 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:34.034 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:34.034 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":351,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:04:44.171062+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:35.861722+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.711473+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:36.661920+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:36.301038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:37.047 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:37.320 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:37.321 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":353,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:37.183975+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.403055+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:37.553943+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.405360+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:39.098766+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.279959+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:38.867624+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:40.333 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:40.602 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:40.602 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":355,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:40.192730+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:41.809585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:42.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.212161+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:04:40.303152+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:43.615 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:43.884 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":358,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:43.208074+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:04:43.232681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:46.896 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:47.166 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:47.166 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:50.179 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:50.453 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:50.453 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:53.466 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:53.733 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:53.734 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:05:56.746 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:05:57.016 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:05:57.017 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:00.030 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:00.298 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:00.298 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:03.311 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:03.581 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:03.581 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:06.594 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:06.865 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:06.866 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:09.879 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:10.149 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:10.150 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:13.161 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:13.432 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:13.432 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:16.445 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:16.714 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:16.714 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:19.727 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:19.994 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:19.995 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:23.007 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:23.277 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:23.278 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:26.290 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:26.559 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:26.559 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:29.571 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:29.839 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:29.840 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:32.852 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:33.126 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:36.138 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:36.408 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:36.409 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":359,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:05:44.212771+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.055733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:38.550810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.917101+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.071492+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:36.707253+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:36.343959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:39.421 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:39.689 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:39.689 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":361,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:39.318419+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.124992+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.466804+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.419842+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:40.041763+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.057250+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:41.210322+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:39.785012+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:42.702 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:42.971 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:42.971 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":364,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:42.329414+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:05:42.302176+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:05:44.110818+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:45.983 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:46.303 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T01:06:46.302+0000 7f07c259c700 0 --2- 172.21.15.148:0/775006333 >> v2:172.21.15.139:3300/0 conn(0x7f07c40ba300 0x7f07c414c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T01:06:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:46.455 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:49.468 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:49.737 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:49.737 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:52.749 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:53.019 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:53.019 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:56.032 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:56.301 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:56.302 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:06:59.314 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:06:59.583 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:06:59.583 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:02.596 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:02.866 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:02.867 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:05.878 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:06.151 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:06.152 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:09.164 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:09.433 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:09.433 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:12.446 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:12.714 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:12.714 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:15.726 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:15.996 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:15.997 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:19.009 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:19.287 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:19.287 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:22.301 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:22.571 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:22.571 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:25.583 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:25.853 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:25.853 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:28.865 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:29.132 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:29.133 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:32.145 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:32.415 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:32.416 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:35.428 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:35.696 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:35.696 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":366,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:06:45.340616+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.342355+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.985464+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:36.936114+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:36.828285+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:38.709 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:38.980 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:38.980 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":368,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:37.751792+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.645247+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:38.590614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.210551+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.829646+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.512931+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:39.245215+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:40.765899+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:41.993 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:42.265 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:42.265 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":371,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:41.772054+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:42.163637+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:43.129000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:06:41.882234+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:45.278 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:45.555 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:45.556 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":373,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:44.205793+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:06:44.489330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:48.568 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:48.839 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:48.839 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:51.854 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:52.123 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:52.124 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:55.136 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:55.407 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:55.407 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:07:58.420 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:07:58.692 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:07:58.693 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:01.705 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:01.975 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:01.976 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:04.988 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:05.256 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:05.256 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:08.268 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:08.537 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:08.538 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:11.550 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:11.819 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:11.820 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:14.832 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:15.102 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:15.102 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:18.115 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:18.387 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:18.387 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:21.400 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:21.671 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:21.672 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:24.685 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:24.953 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:24.953 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:27.966 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:28.237 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:28.238 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:31.250 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:31.521 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:31.522 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:34.535 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:34.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:34.802 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:37.815 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:38.086 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:38.087 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":374,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:07:46.214858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.856022+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.666949+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.397310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.017549+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:37.288957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:39.279765+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:37.228764+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:36.913830+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:41.099 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:41.373 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:41.374 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":377,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:41.215014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.051433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:43.231131+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.752829+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:42.321628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:41.559475+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:07:40.837849+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:44.387 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:44.657 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:44.684 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":380,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:44.228900+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:07:45.479727+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:47.687 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:47.957 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:47.958 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:50.970 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:51.236 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:51.237 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:54.249 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:54.519 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:54.520 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:08:57.532 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:08:57.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:08:57.802 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:00.815 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:01.085 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:01.086 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:04.099 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:04.367 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:04.368 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:07.381 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:07.650 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:07.651 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:10.663 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:10.931 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:10.931 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:13.944 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:14.214 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:14.214 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:17.228 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:17.500 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:17.501 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:20.514 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:20.784 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:20.784 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:23.797 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:24.069 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:24.069 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:27.081 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:27.348 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:27.348 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:30.361 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:30.630 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:30.631 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:33.643 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:33.915 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:33.916 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:36.929 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:37.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:37.199 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":381,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:08:46.236600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.929215+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.632857+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.368470+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:39.468017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:37.313971+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:37.808424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:40.211 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:40.480 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:40.481 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":383,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:40.362240+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:40.223397+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.138574+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.035524+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.630877+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:41.629376+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:43.494 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:43.761 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:43.762 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":386,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:43.374115+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.244082+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:43.422202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:08:45.550080+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:08:42.744948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:46.774 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:47.046 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:47.047 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:50.059 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:50.329 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:50.329 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:53.342 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:53.613 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:53.614 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:56.626 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:09:56.894 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:09:56.895 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:09:59.907 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:00.173 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:00.174 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:03.187 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:03.457 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:03.458 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:06.470 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:06.737 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:06.738 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:09.750 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:10.020 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:10.020 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:13.033 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:13.304 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:13.304 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:16.316 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:16.584 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:19.597 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:19.863 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:19.863 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:22.876 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:23.144 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:23.144 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:26.157 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:26.427 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:26.427 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:29.440 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:29.713 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:29.713 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:32.725 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:32.993 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:32.993 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:36.006 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:36.282 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:36.282 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":388,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:09:46.384391+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.309879+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:37.970890+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:37.434002+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:38.204977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:39.295 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:39.567 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:39.567 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":390,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:39.370368+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.288803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:40.468828+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:39.689095+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:42.579 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:42.850 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:42.851 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":391,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:41.382945+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:44.184538+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.610825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.084285+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.632242+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:43.180481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.623860+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:09:42.635077+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:45.864 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:46.134 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:46.135 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":394,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:45.394832+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:09:45.833855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:49.149 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:49.422 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:49.422 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:52.434 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:52.704 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:55.716 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:55.983 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:10:58.995 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:10:59.267 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:10:59.267 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:02.280 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:02.548 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:05.561 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:05.829 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:05.830 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:08.842 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:09.114 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:09.115 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:12.127 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:12.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:12.397 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:15.410 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:15.682 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:15.682 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:18.695 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:18.965 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:18.966 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:21.980 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:22.251 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:22.252 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:25.264 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:25.531 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:25.531 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:28.544 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:28.812 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:28.812 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:31.827 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:32.096 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:32.096 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:35.108 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:35.375 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:35.375 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:38.388 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:38.656 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:38.657 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":395,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:10:46.399705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.489155+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.990056+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:37.553296+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:38.937824+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:41.669 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:41.938 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:41.939 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":397,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:40.394264+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.567520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.734806+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.687917+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.752260+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.296181+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:42.626345+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:40.655289+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:44.951 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:45.221 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:45.221 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":399,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:44.240631+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.696961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:44.460367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:10:45.962326+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:10:43.435202+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:48.235 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:48.509 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:48.509 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:51.521 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:51.790 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:51.790 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:54.804 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:55.077 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:11:58.088 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:11:58.357 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:11:58.358 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:01.370 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:01.638 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:01.639 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:04.651 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:04.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:04.921 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:07.934 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:08.204 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:08.204 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:11.216 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:11.487 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:11.487 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:14.500 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:14.768 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:14.768 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:17.781 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:18.049 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:18.050 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:21.062 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:21.338 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:21.338 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:24.350 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:24.619 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:24.620 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:27.633 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:27.901 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:27.902 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:30.915 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:31.185 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:31.185 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:34.198 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:34.468 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:34.468 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:37.481 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:37.749 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:37.750 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":402,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:11:47.255351+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.539887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:39.931551+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:38.407660+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:38.957934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:40.762 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:41.032 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:41.032 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":404,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:40.406497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.524590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.522357+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:41.440974+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:44.044 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:44.312 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:44.313 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":406,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:43.412754+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:45.248984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.572453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.349394+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.984445+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.140917+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:43.558082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:11:44.433567+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:47.325 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:47.595 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:47.595 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":408,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:45.424666+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:11:46.786135+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:50.608 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:50.875 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:50.876 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:53.888 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:54.158 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:54.158 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:12:57.172 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:12:57.445 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:12:57.445 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:00.457 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:00.728 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:00.729 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:03.741 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:04.010 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:04.011 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:07.023 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:07.293 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:07.293 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:10.305 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:10.576 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:10.576 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:13.589 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:13.858 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:13.858 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:16.871 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:17.140 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:17.140 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:20.153 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:20.420 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:20.421 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:23.434 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:23.704 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:23.705 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:26.717 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:26.987 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:26.988 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:30.000 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:30.269 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:30.270 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:33.283 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:33.552 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:33.553 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:36.565 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:36.834 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:36.834 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":409,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:12:48.435317+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:39.093526+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:39.847 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:40.118 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:40.118 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":410,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:39.771516+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.371828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.430019+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:42.494220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.951413+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:41.736117+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:39.131821+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:43.131 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:43.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:43.402 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":412,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:43.267005+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.320232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.092563+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.150754+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.584294+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.566663+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:44.559122+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:46.414 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:46.685 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:46.685 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":414,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:46.276140+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:12:45.357721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:12:47.637669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:49.698 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:49.968 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:49.968 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:52.981 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:53.249 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:53.250 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:56.262 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:56.533 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:56.533 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:13:59.545 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:13:59.814 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:13:59.815 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:02.828 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:03.095 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:03.095 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:06.108 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:06.377 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:06.377 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:09.390 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:09.661 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:09.661 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:12.673 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:12.943 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:12.943 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:15.956 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:16.225 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:16.226 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:19.239 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:19.505 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:19.505 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:22.518 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:22.787 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:22.787 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:25.799 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:26.119 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T01:14:26.118+0000 7f372f7fe700 0 --2- 172.21.15.148:0/3024221136 >> v2:172.21.15.50:3300/0 conn(0x7f3730147790 0x7f37300b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T01:14:26.270 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:26.271 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:29.284 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:29.553 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:29.554 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:32.566 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:32.835 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:32.853 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:35.856 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:36.126 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:36.127 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:39.141 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:39.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:39.410 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":416,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:13:48.285261+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.056570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.020006+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:39.720942+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:40.103810+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:42.423 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:42.693 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:42.693 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":418,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:41.433570+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:43.054993+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.699067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.026965+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:42.371963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:44.654796+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:45.706 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:45.979 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:45.980 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":421,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:45.445493+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.142708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:46.307605+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.076518+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.206419+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:13:45.341540+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:48.992 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:49.262 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:49.262 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":423,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:48.277775+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:13:47.945039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:52.274 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:52.542 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:52.542 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:55.554 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:55.823 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:55.824 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:14:58.837 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:14:59.107 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:14:59.107 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:02.120 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:02.393 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:02.394 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:05.406 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:05.675 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:05.675 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:08.688 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:08.957 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:08.958 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:11.972 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:12.243 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:12.244 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:15.256 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:15.525 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:15.526 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:18.538 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:18.805 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:18.805 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:21.818 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:22.089 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:22.089 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:25.102 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:25.373 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:25.374 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:28.386 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:28.657 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:28.658 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:31.671 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:31.941 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:31.942 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:34.954 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:35.225 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:35.225 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:38.238 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:38.509 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:38.510 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":424,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:14:49.455867+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.421347+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.155365+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:39.729998+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:40.756589+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:41.523 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:41.796 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:41.796 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":426,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:41.454858+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.811913+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:43.672889+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:42.431479+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:44.809 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:45.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:45.079 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":428,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:44.462192+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.252153+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.458449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.060586+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.203325+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:46.204868+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:45.331070+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:48.091 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:48.363 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:48.363 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":431,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:47.474233+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:14:47.329526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:14:48.728563+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:51.375 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:51.643 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:51.644 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:54.656 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:54.926 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:54.927 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:15:57.939 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:15:58.209 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:15:58.210 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:01.223 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:01.490 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:01.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:04.503 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:04.774 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:04.775 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:07.787 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:08.055 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:08.056 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:11.068 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:11.338 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:11.338 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:14.351 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:14.623 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:14.624 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:17.636 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:17.907 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:17.908 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:20.921 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:21.186 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:21.187 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:24.200 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:24.469 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:24.470 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:27.482 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:27.752 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:27.752 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:30.765 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:31.033 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:31.033 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:34.046 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:34.315 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:34.316 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:37.328 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:37.598 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:37.598 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":433,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:15:49.483658+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.319852+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:40.611 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:40.880 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:40.880 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":434,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:40.386439+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:40.813608+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:40.493336+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:41.319356+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:43.892 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:44.161 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:44.162 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":436,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:42.392580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.894150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:43.941682+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:44.443001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.471290+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.796820+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.037665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:42.683065+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:45.399105+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:47.175 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:47.446 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:47.446 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":440,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:47.418426+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:47.688861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:15:48.771184+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:15:46.631954+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:50.460 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:50.730 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:50.731 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:53.743 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:54.012 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:54.012 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:16:57.024 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:16:57.295 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:16:57.295 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:00.307 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:00.579 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:00.579 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:03.592 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:03.861 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:03.862 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:06.874 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:07.146 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:07.146 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:10.158 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:10.428 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:10.428 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:13.441 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:13.710 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:13.710 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:16.722 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:17.063 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:17.063 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:20.076 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:20.344 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:20.344 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:23.357 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:23.634 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:23.634 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:26.647 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:26.916 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:26.917 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:29.929 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:30.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:30.199 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:33.211 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:33.479 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:33.480 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:36.492 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:36.759 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:36.759 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:39.772 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:40.040 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:40.041 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":442,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:16:49.428762+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.822032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:40.335400+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:41.145849+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:41.557133+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:43.053 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:43.322 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":444,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:42.096690+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.989974+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.420738+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.059698+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:43.545919+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:45.406010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:46.334 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:46.601 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:46.602 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":447,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:46.317684+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.172968+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:48.062526+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:16:48.844304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:46.576475+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.267978+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:16:47.589532+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:49.614 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:49.883 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:49.883 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:52.896 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:53.162 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:53.163 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:56.175 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:56.446 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:56.447 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:17:59.459 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:17:59.729 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:17:59.729 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:02.741 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:03.010 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:03.011 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:06.022 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:06.288 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:06.289 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:09.303 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:09.574 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:09.575 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:12.587 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:12.856 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:12.857 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:15.870 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:16.136 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:16.137 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:19.150 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:19.421 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:19.422 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:22.434 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:22.703 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:22.703 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:25.716 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:25.984 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:25.985 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:28.997 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:29.267 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:29.268 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:32.281 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:32.551 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:32.551 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:35.564 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:35.832 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:35.833 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:38.845 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:39.114 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:39.114 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":449,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:17:49.336497+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:40.912975+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:42.127 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:42.398 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:42.398 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":450,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:41.484346+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.177860+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.688251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:44.531625+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:42.079017+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:41.999258+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:45.411 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:45.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:45.682 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":453,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:45.498820+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.144338+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:45.511440+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.443040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.309378+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.108026+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:47.937774+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:46.244635+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:48.694 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:48.962 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:48.963 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":456,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:48.509169+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:17:48.979250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:17:49.046885+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:51.976 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:52.247 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:52.247 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:55.260 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:55.530 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:55.531 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:18:58.543 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:18:58.814 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:18:58.814 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:01.826 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:02.096 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:02.096 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:05.109 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:05.378 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:05.379 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:08.391 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:08.663 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:08.664 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:11.676 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:11.949 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:11.949 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:14.962 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:15.231 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:15.232 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:18.244 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:18.517 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:18.518 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:21.531 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:21.801 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:21.801 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:24.813 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:25.083 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:25.084 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:28.096 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:28.368 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:28.369 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:31.381 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:31.651 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:31.651 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:34.664 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:34.933 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:34.933 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:37.946 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:38.217 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:38.218 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:41.231 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:41.497 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:41.498 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":458,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:18:50.517580+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.144342+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:41.041198+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:42.821999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:42.200673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:44.510 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:44.783 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:44.783 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":461,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:44.506705+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.141856+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:45.971899+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.557225+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:44.640070+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:46.526137+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:47.796 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:48.065 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:48.066 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":464,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:47.519582+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.348370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:49.155663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:18:49.903907+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.460952+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.258030+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:18:48.404914+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:51.078 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:51.349 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:51.350 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:54.362 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:54.632 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:54.632 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:19:57.645 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:19:57.917 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:19:57.917 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:00.930 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:01.198 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:01.198 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:04.211 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:04.481 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:04.482 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:07.495 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:07.761 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:07.762 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:10.774 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:11.043 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:11.043 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:14.055 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:14.327 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:14.327 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:17.339 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:17.607 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:17.607 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:20.620 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:20.889 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:20.889 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:23.901 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:24.169 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:24.169 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:27.182 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:27.450 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:27.450 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:30.463 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:30.735 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:30.736 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:33.748 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:34.020 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:34.021 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:37.032 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:37.302 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:37.303 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:40.315 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:40.584 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:40.585 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":467,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:19:50.541748+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:43.053070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:41.890013+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:42.332620+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:43.598 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:43.867 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:43.868 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":468,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:43.336873+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.461050+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:43.503489+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:46.880 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:47.152 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:47.153 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":470,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:46.353584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.571590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.550156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.769826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:45.979816+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.321395+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.582354+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:49.058574+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:48.522628+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:19:46.568260+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:50.165 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:50.436 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:50.437 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":473,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:50.364573+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:19:50.027729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:53.449 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:53.717 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:53.717 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:20:56.730 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:20:57.004 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:20:57.005 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:00.018 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:00.287 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:00.287 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:03.300 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:03.568 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:03.568 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:06.580 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:06.848 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:06.849 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:09.861 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:10.129 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:10.129 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:13.142 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:13.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:16.422 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:16.692 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:16.693 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:19.706 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:19.977 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:19.978 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:22.990 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:23.258 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:23.258 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:26.270 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:26.538 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:26.539 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:29.551 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:29.825 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:32.837 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:33.107 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:33.108 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:36.120 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:36.399 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:36.399 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:39.412 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:39.682 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:39.683 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:42.695 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:42.964 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:42.965 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":474,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:20:51.369854+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.143939+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:42.377492+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:43.654883+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:43.131156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:45.978 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:46.246 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:46.247 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":476,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:44.360429+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.497078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.762948+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.642347+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:45.478697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:46.606496+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:49.259 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:49.530 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:49.531 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":479,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:48.382957+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.511896+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:50.059171+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:20:50.940851+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.470449+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.101074+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:20:49.493757+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:52.543 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:52.812 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:52.812 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:55.824 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:56.094 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:56.095 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:21:59.107 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:21:59.379 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:21:59.380 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:02.392 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:02.665 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:02.665 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:05.677 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:05.947 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:05.948 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:08.961 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:09.229 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:09.230 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:12.242 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:12.518 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:12.519 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:15.523 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:15.792 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:15.793 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:18.804 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:19.077 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:19.078 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:22.090 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:22.367 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:22.368 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:25.375 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:25.651 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:25.652 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:28.664 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:28.941 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:31.947 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:32.217 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:32.217 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:35.229 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:35.500 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:35.500 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:38.512 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:38.785 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:38.785 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:41.798 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:42.065 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:42.066 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":481,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:21:51.396324+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.851398+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.237860+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:43.744245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:43.279844+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:45.078 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:45.350 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:45.351 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":482,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:44.364905+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.096401+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.631455+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:47.252436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:45.685405+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:46.689164+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:48.364 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:48.633 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:48.633 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":485,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:48.380498+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.161413+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:50.272816+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.496528+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.954837+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:21:49.792478+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:51.645 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:51.915 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:51.916 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":487,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:51.391030+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:21:51.178644+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:54.928 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:55.195 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:55.195 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:22:58.208 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:22:58.478 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:22:58.478 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:01.491 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:01.760 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:01.761 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:04.773 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:05.045 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:05.046 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:08.058 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:08.326 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:08.327 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:11.339 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:11.608 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:11.608 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:14.620 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:14.893 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:14.894 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:17.906 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:18.175 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:18.175 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:21.187 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:21.455 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:21.455 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:24.468 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:24.736 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:24.737 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:27.749 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:28.019 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:28.019 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:31.031 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:31.302 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:31.303 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:34.315 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:34.586 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:34.587 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:37.599 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:37.868 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:37.868 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:40.881 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:41.150 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:41.151 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":488,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:22:52.394434+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:43.371939+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:44.163 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:44.433 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:44.433 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":489,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:44.376600+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.016509+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.325712+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:44.073853+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:44.270700+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:47.446 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:47.714 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:47.715 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":491,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:47.384232+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.043192+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:47.816585+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:48.200647+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:46.787980+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:50.727 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:50.995 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:50.996 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":493,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:49.391766+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.448976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.944367+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:22:51.985697+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.332884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:50.478563+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:22:49.934527+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:54.008 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:54.280 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:54.280 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:23:57.293 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:23:57.563 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:23:57.564 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:00.577 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:00.843 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:00.844 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:03.856 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:04.124 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:04.125 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:07.138 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:07.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:07.410 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:10.424 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:10.697 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:10.698 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:13.711 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:13.980 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:13.981 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:16.993 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:17.261 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:17.261 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:20.273 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:20.542 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:20.542 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:23.554 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:23.820 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:23.821 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:26.833 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:27.103 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:27.104 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:30.116 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:30.384 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:30.384 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:33.397 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:33.669 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:33.669 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:36.682 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:36.952 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:36.952 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:39.965 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:40.232 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:40.233 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:43.245 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:43.515 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:43.515 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":495,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:23:53.402925+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.609882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.264166+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:44.604836+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:44.943068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:46.527 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:46.796 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:46.797 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":497,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:46.392756+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.725586+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.190697+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:47.564490+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:49.810 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:50.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:50.079 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":499,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:49.408266+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.388509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.134742+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.946660+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:51.182755+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:48.984564+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.593040+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:23:50.853927+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:53.091 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:53.361 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:53.362 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":502,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:52.426470+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:23:52.765050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:56.374 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:56.644 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:56.645 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:24:59.658 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:24:59.927 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:24:59.928 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:02.940 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:03.210 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:03.211 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:06.223 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:06.494 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:06.495 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:09.507 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:09.777 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:09.778 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:12.790 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:13.060 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:13.060 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:16.072 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:16.392 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T01:25:16.393+0000 7f461d007700 0 --2- 172.21.15.148:0/3935122396 >> v2:172.21.15.148:3300/0 conn(0x7f46180ba300 0x7f46180b7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T01:25:16.545 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:16.546 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:19.558 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:19.828 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:19.829 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:22.842 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:23.110 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:23.160 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:26.163 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:26.433 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:26.434 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:29.446 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:29.715 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:29.715 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:32.728 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:32.995 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:32.996 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:36.008 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:36.274 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:36.275 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:39.288 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:39.558 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:39.559 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:42.572 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:42.842 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:42.842 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":503,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:24:53.431802+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.647957+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:44.954358+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:45.854 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:46.123 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":504,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:45.586594+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:45.525453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:45.646721+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:49.136 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:49.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:49.403 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":505,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:46.588430+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.525330+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.205550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.802499+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.902267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:49.806753+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.686004+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:50.945662+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.143644+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:48.568744+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:52.415 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:52.688 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:52.689 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":509,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:52.610789+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:24:53.361997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:24:51.999740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:55.701 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:55.970 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:55.970 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:25:58.983 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:25:59.252 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:25:59.253 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:02.267 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:02.536 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:02.536 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:05.549 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:05.816 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:05.817 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:08.830 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:09.096 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:09.096 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:12.109 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:12.376 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:12.377 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:15.390 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:15.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:15.660 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:18.672 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:18.943 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:18.943 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:21.956 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:22.223 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:22.224 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:25.236 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:25.558 INFO:teuthology.orchestra.run.smithi148.stderr:2020-03-09T01:26:25.559+0000 7f9dcab48700 0 --2- 172.21.15.148:0/1502476350 >> v2:172.21.15.50:3300/0 conn(0x7f9dc41421c0 0x7f9dc40b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2020-03-09T01:26:25.706 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:25.707 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:28.720 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:28.988 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:28.989 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:32.001 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:32.283 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:35.297 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:35.566 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:35.567 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:38.580 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:38.856 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:38.857 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:41.869 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:42.139 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:42.139 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:45.151 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:45.422 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:45.422 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":510,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:25:53.614740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.665956+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:44.693060+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:45.819397+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:45.113689+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:48.435 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:48.706 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:48.706 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":512,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:46.606114+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.304534+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.511461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.528574+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.876953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:50.754931+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:48.671963+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:49.530366+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:51.719 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:51.988 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:51.989 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":515,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:51.625255+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.177982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:25:53.432499+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:52.682219+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:25:51.781472+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:55.002 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:55.270 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:55.271 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:26:58.283 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:26:58.551 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:26:58.551 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:01.563 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:01.830 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:01.831 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:04.843 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:05.110 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:05.111 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:08.123 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:08.394 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:08.394 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:11.407 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:11.675 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:11.675 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:14.687 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:14.957 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:14.957 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:17.970 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:18.235 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:18.236 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:21.247 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:21.518 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:21.518 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:24.531 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:24.801 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:24.801 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:27.814 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:28.088 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:28.089 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:31.100 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:31.371 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:31.371 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:34.385 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:34.655 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:34.655 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:37.667 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:37.939 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:37.940 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:40.951 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:41.220 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:41.221 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:44.235 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:44.504 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:44.504 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":517,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:26:53.635136+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:46.386708+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.321688+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:46.090847+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:45.344666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:47.517 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:47.787 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:47.788 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":519,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:47.560721+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.430289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:49.572874+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:50.800 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:51.068 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:51.069 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":520,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:50.536726+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.115554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.419066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.287290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.621665+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.046958+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:51.836673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:50.451827+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:54.081 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:54.349 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:54.350 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":523,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:53.547792+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:26:53.570853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:26:53.048609+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:27:57.363 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:27:57.631 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:27:57.632 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:00.644 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:00.914 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:00.914 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:03.929 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:04.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:04.199 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:07.212 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:07.483 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:07.483 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:10.496 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:10.767 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:10.767 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:13.780 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:14.050 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:14.051 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:17.063 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:17.332 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:17.333 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:20.345 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:20.614 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:20.615 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:23.629 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:23.899 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:23.899 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:26.911 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:27.181 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:27.181 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:30.194 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:30.463 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:30.463 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:33.476 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:33.746 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:33.746 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:36.759 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:37.029 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:37.029 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:40.041 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:40.310 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:40.310 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:43.323 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:43.592 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:43.593 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":524,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:27:54.552584+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.102178+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:46.605 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:46.880 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:46.880 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":525,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:46.428262+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:47.226586+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:46.177351+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:46.110477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:49.894 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:50.164 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:50.165 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":526,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:47.430338+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.582299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.575901+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.530795+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.581252+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:51.185427+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.485659+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:50.618167+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:53.178 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:53.448 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":528,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:52.446799+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.457590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:27:54.149897+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:53.964548+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:27:52.650511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:56.460 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:28:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:28:56.732 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:28:59.744 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:00.012 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:00.012 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:03.025 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:03.294 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:03.294 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:06.307 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:06.576 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:06.577 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:09.589 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:09.856 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:09.857 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:12.870 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:13.140 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:13.141 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:16.153 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:16.425 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:16.426 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:19.438 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:19.708 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:19.708 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:22.721 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:22.989 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:22.990 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:26.002 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:26.271 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:26.272 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:29.284 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:29.554 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:29.555 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:32.567 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:32.836 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:32.836 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:35.849 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:36.115 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:36.116 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:39.128 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:39.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:39.397 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:42.409 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:42.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:42.679 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:45.692 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:45.962 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:45.963 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":530,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:28:55.459014+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:47.301998+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.289337+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:47.144194+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:46.503879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:48.976 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:49.246 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:49.247 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":532,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:48.019740+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.830445+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.582104+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:50.604935+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.203512+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:52.260 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:52.531 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:52.532 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":534,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:52.038386+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.309177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:51.727461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:28:54.803617+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:54.952869+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.395572+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:52.034863+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:28:53.589828+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:55.545 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:55.814 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:55.814 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:29:58.826 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:29:59.095 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:29:59.095 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:02.108 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:02.380 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:02.380 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:05.393 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:05.662 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:05.663 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:08.674 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:08.944 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:08.945 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:11.957 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:12.228 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:12.229 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:15.242 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:15.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:15.513 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:18.526 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:18.797 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:18.797 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:21.810 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:22.083 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:22.084 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:25.096 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:25.365 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:25.366 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:28.378 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:28.649 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:28.650 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:31.660 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:31.929 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:31.929 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:34.942 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:35.211 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:35.212 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:38.224 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:38.494 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:38.494 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:41.507 INFO:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-03-09T01:30:41.777 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-03-09T01:30:41.777 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":537,"fsid":"036b76b6-511e-4ec6-9669-0a47550bd261","created":"2020-03-09T00:24:06.736934+0000","modified":"2020-03-09T01:29:55.455095+0000","last_up_change":"2020-03-09T00:25:16.760245+0000","last_in_change":"2020-03-09T00:25:00.770574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-03-09T00:24:09.667386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-03-09T00:24:26.419136+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d0be76d1-3ce6-4c23-9917-41b35482def3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":27385},{"type":"v1","addr":"172.21.15.148:6817","nonce":27385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":27385},{"type":"v1","addr":"172.21.15.148:6819","nonce":27385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":27385},{"type":"v1","addr":"172.21.15.148:6824","nonce":27385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":27385},{"type":"v1","addr":"172.21.15.148:6821","nonce":27385}]},"public_addr":"172.21.15.148:6817/27385","cluster_addr":"172.21.15.148:6819/27385","heartbeat_back_addr":"172.21.15.148:6824/27385","heartbeat_front_addr":"172.21.15.148:6821/27385","state":["exists","up"]},{"osd":1,"uuid":"cee70020-657e-4b62-89cb-226f760442f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":27629},{"type":"v1","addr":"172.21.15.50:6809","nonce":27629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":27629},{"type":"v1","addr":"172.21.15.50:6811","nonce":27629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":27629},{"type":"v1","addr":"172.21.15.50:6815","nonce":27629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":27629},{"type":"v1","addr":"172.21.15.50:6813","nonce":27629}]},"public_addr":"172.21.15.50:6809/27629","cluster_addr":"172.21.15.50:6811/27629","heartbeat_back_addr":"172.21.15.50:6815/27629","heartbeat_front_addr":"172.21.15.50:6813/27629","state":["exists","up"]},{"osd":2,"uuid":"4103b4d7-7a38-48e4-8403-3549f5b3fda2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":26980},{"type":"v1","addr":"172.21.15.139:6801","nonce":26980}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":26980},{"type":"v1","addr":"172.21.15.139:6803","nonce":26980}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":26980},{"type":"v1","addr":"172.21.15.139:6807","nonce":26980}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":26980},{"type":"v1","addr":"172.21.15.139:6805","nonce":26980}]},"public_addr":"172.21.15.139:6801/26980","cluster_addr":"172.21.15.139:6803/26980","heartbeat_back_addr":"172.21.15.139:6807/26980","heartbeat_front_addr":"172.21.15.139:6805/26980","state":["exists","up"]},{"osd":3,"uuid":"ce80df27-1b2e-41e1-9ded-a012ad488591","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":28012},{"type":"v1","addr":"172.21.15.104:6803","nonce":28012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":28012},{"type":"v1","addr":"172.21.15.104:6805","nonce":28012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":28012},{"type":"v1","addr":"172.21.15.104:6809","nonce":28012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":28012},{"type":"v1","addr":"172.21.15.104:6807","nonce":28012}]},"public_addr":"172.21.15.104:6803/28012","cluster_addr":"172.21.15.104:6805/28012","heartbeat_back_addr":"172.21.15.104:6809/28012","heartbeat_front_addr":"172.21.15.104:6807/28012","state":["exists","up"]},{"osd":4,"uuid":"8ff5a871-6f10-4e55-ac60-33bdc11c0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":27365},{"type":"v1","addr":"172.21.15.148:6801","nonce":27365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":27365},{"type":"v1","addr":"172.21.15.148:6803","nonce":27365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":27365},{"type":"v1","addr":"172.21.15.148:6807","nonce":27365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":27365},{"type":"v1","addr":"172.21.15.148:6805","nonce":27365}]},"public_addr":"172.21.15.148:6801/27365","cluster_addr":"172.21.15.148:6803/27365","heartbeat_back_addr":"172.21.15.148:6807/27365","heartbeat_front_addr":"172.21.15.148:6805/27365","state":["exists","up"]},{"osd":5,"uuid":"5122cc94-7217-4d84-8b81-11383b48c416","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":27630},{"type":"v1","addr":"172.21.15.50:6801","nonce":27630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":27630},{"type":"v1","addr":"172.21.15.50:6803","nonce":27630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":27630},{"type":"v1","addr":"172.21.15.50:6807","nonce":27630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":27630},{"type":"v1","addr":"172.21.15.50:6805","nonce":27630}]},"public_addr":"172.21.15.50:6801/27630","cluster_addr":"172.21.15.50:6803/27630","heartbeat_back_addr":"172.21.15.50:6807/27630","heartbeat_front_addr":"172.21.15.50:6805/27630","state":["exists","up"]},{"osd":6,"uuid":"1bea4f99-5e2a-4fbc-bac3-8cb45d678742","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":54,"down_at":67,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":26979},{"type":"v1","addr":"172.21.15.139:6811","nonce":26979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":26979},{"type":"v1","addr":"172.21.15.139:6815","nonce":26979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6823","nonce":26979},{"type":"v1","addr":"172.21.15.139:6826","nonce":26979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6817","nonce":26979},{"type":"v1","addr":"172.21.15.139:6820","nonce":26979}]},"public_addr":"172.21.15.139:6811/26979","cluster_addr":"172.21.15.139:6815/26979","heartbeat_back_addr":"172.21.15.139:6826/26979","heartbeat_front_addr":"172.21.15.139:6820/26979","state":["exists"]},{"osd":7,"uuid":"f6a853dd-1462-4f8c-83d6-701258d78519","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":28042},{"type":"v1","addr":"172.21.15.104:6819","nonce":28042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":28042},{"type":"v1","addr":"172.21.15.104:6822","nonce":28042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6827","nonce":28042},{"type":"v1","addr":"172.21.15.104:6829","nonce":28042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":28042},{"type":"v1","addr":"172.21.15.104:6826","nonce":28042}]},"public_addr":"172.21.15.104:6819/28042","cluster_addr":"172.21.15.104:6822/28042","heartbeat_back_addr":"172.21.15.104:6829/28042","heartbeat_front_addr":"172.21.15.104:6826/28042","state":["exists","up"]},{"osd":8,"uuid":"b9911260-c1d0-4563-892b-35a9e1c47a9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":27364},{"type":"v1","addr":"172.21.15.148:6809","nonce":27364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":27364},{"type":"v1","addr":"172.21.15.148:6811","nonce":27364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":27364},{"type":"v1","addr":"172.21.15.148:6815","nonce":27364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":27364},{"type":"v1","addr":"172.21.15.148:6813","nonce":27364}]},"public_addr":"172.21.15.148:6809/27364","cluster_addr":"172.21.15.148:6811/27364","heartbeat_back_addr":"172.21.15.148:6815/27364","heartbeat_front_addr":"172.21.15.148:6813/27364","state":["exists","up"]},{"osd":9,"uuid":"d7ebf4f6-a2b8-4076-83dd-504172481e32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6821","nonce":27643},{"type":"v1","addr":"172.21.15.50:6823","nonce":27643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6825","nonce":27643},{"type":"v1","addr":"172.21.15.50:6827","nonce":27643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":27643},{"type":"v1","addr":"172.21.15.50:6831","nonce":27643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":27643},{"type":"v1","addr":"172.21.15.50:6829","nonce":27643}]},"public_addr":"172.21.15.50:6823/27643","cluster_addr":"172.21.15.50:6827/27643","heartbeat_back_addr":"172.21.15.50:6831/27643","heartbeat_front_addr":"172.21.15.50:6829/27643","state":["exists","up"]},{"osd":10,"uuid":"cb76df3e-6ea9-4ee6-8b03-84e3cd329c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":26978},{"type":"v1","addr":"172.21.15.139:6810","nonce":26978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":26978},{"type":"v1","addr":"172.21.15.139:6814","nonce":26978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":26978},{"type":"v1","addr":"172.21.15.139:6825","nonce":26978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":26978},{"type":"v1","addr":"172.21.15.139:6819","nonce":26978}]},"public_addr":"172.21.15.139:6810/26978","cluster_addr":"172.21.15.139:6814/26978","heartbeat_back_addr":"172.21.15.139:6825/26978","heartbeat_front_addr":"172.21.15.139:6819/26978","state":["exists","up"]},{"osd":11,"uuid":"8bda27ba-cf17-472c-9a21-02dcaefab68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":28013},{"type":"v1","addr":"172.21.15.104:6811","nonce":28013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":28013},{"type":"v1","addr":"172.21.15.104:6813","nonce":28013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":28013},{"type":"v1","addr":"172.21.15.104:6817","nonce":28013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":28013},{"type":"v1","addr":"172.21.15.104:6815","nonce":28013}]},"public_addr":"172.21.15.104:6811/28013","cluster_addr":"172.21.15.104:6813/28013","heartbeat_back_addr":"172.21.15.104:6817/28013","heartbeat_front_addr":"172.21.15.104:6815/28013","state":["exists","up"]},{"osd":12,"uuid":"d861da04-0be8-434b-9ed8-c1ec97fca7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6823","nonce":27395},{"type":"v1","addr":"172.21.15.148:6825","nonce":27395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":27395},{"type":"v1","addr":"172.21.15.148:6827","nonce":27395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":27395},{"type":"v1","addr":"172.21.15.148:6831","nonce":27395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":27395},{"type":"v1","addr":"172.21.15.148:6829","nonce":27395}]},"public_addr":"172.21.15.148:6825/27395","cluster_addr":"172.21.15.148:6827/27395","heartbeat_back_addr":"172.21.15.148:6831/27395","heartbeat_front_addr":"172.21.15.148:6829/27395","state":["exists","up"]},{"osd":13,"uuid":"4ba3410e-df5b-4529-9ebe-34c15f68bc9d","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":27631},{"type":"v1","addr":"172.21.15.50:6817","nonce":27631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":27631},{"type":"v1","addr":"172.21.15.50:6819","nonce":27631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":27631},{"type":"v1","addr":"172.21.15.50:6826","nonce":27631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":27631},{"type":"v1","addr":"172.21.15.50:6822","nonce":27631}]},"public_addr":"172.21.15.50:6817/27631","cluster_addr":"172.21.15.50:6819/27631","heartbeat_back_addr":"172.21.15.50:6826/27631","heartbeat_front_addr":"172.21.15.50:6822/27631","state":["exists","up"]},{"osd":14,"uuid":"1ba69402-6d5c-491e-ac86-747517533a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":26981},{"type":"v1","addr":"172.21.15.139:6821","nonce":26981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":26981},{"type":"v1","addr":"172.21.15.139:6827","nonce":26981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":26981},{"type":"v1","addr":"172.21.15.139:6831","nonce":26981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":26981},{"type":"v1","addr":"172.21.15.139:6829","nonce":26981}]},"public_addr":"172.21.15.139:6821/26981","cluster_addr":"172.21.15.139:6827/26981","heartbeat_back_addr":"172.21.15.139:6831/26981","heartbeat_front_addr":"172.21.15.139:6829/26981","state":["exists","up"]},{"osd":15,"uuid":"a3566380-0c4a-403f-8533-b2d0dcfa8506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6821","nonce":28043},{"type":"v1","addr":"172.21.15.104:6823","nonce":28043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6825","nonce":28043},{"type":"v1","addr":"172.21.15.104:6828","nonce":28043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":28043},{"type":"v1","addr":"172.21.15.104:6833","nonce":28043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":28043},{"type":"v1","addr":"172.21.15.104:6831","nonce":28043}]},"public_addr":"172.21.15.104:6823/28043","cluster_addr":"172.21.15.104:6828/28043","heartbeat_back_addr":"172.21.15.104:6833/28043","heartbeat_front_addr":"172.21.15.104:6831/28043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.541659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.955175+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.499126+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:47.459448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.482520+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:54.966753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T00:24:19.085042+0000","dead_epoch":67},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:54.967972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.241850+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:52.986728+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:53.644814+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.611386+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:50.820971+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":65536,"last_purged_snaps_scrub":"2020-03-09T01:29:47.384553+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:51.314941+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-03-09T01:29:46.521638+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":19,"length":1},{"begin":26,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":1},{"begin":42,"length":3},{"begin":50,"length":2},{"begin":53,"length":1},{"begin":59,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-09T01:30:42.567 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-09T01:30:42.567 DEBUG:teuthology.task.console_log:Killing console logger for smithi139 2020-03-09T01:30:42.567 DEBUG:teuthology.task.console_log:Killing console logger for smithi148 2020-03-09T01:30:42.567 DEBUG:teuthology.task.console_log:Killing console logger for smithi050 2020-03-09T01:30:42.568 DEBUG:teuthology.task.console_log:Killing console logger for smithi104 2020-03-09T01:30:42.568 DEBUG:teuthology.task.console_log:Killing console logger for smithi139 2020-03-09T01:30:42.568 DEBUG:teuthology.task.console_log:Killing console logger for smithi050 2020-03-09T01:30:42.569 DEBUG:teuthology.task.console_log:Killing console logger for smithi148 2020-03-09T01:30:42.569 DEBUG:teuthology.task.console_log:Killing console logger for smithi104 2020-03-09T01:30:42.569 DEBUG:teuthology.exit:Finished running handlers